Faster way to execute my loop

  • (4 Pages)
  • +
  • 1
  • 2
  • 3
  • 4

46 Replies - 1580 Views - Last Post: 28 June 2013 - 10:32 AM Rate Topic: -----

#31 C.Andrews  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 14
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Faster way to execute my loop

Posted 27 June 2013 - 06:39 AM

Here is my entire project for anyone interested. You'll want to change the name of the sample file I previously uploaded to a .gen extension (which the program expects) from .txt (for some reason the forum wouldn't let me attach it as .gen).

The open file dialog box you get when selecting the input file defaults to the application startup directory. Currently, there are two very similar subs in the body of the program, one commented out and the other active; I've been using these to test different variations of loops to determine completion times.

The last 8 lines of the output file will have the run times for each segment in milliseconds. Currently the project is set to compile as a 64-bit application, but you can easily change that in the configuration manager.

Attached File(s)


Was This Post Helpful? 0
  • +
  • -

#32 dbasnett  Icon User is offline

  • D.I.C Addict
  • member icon

Reputation: 109
  • View blog
  • Posts: 603
  • Joined: 01-October 08

Re: Faster way to execute my loop

Posted 27 June 2013 - 07:20 AM

Using the data file from post #6, and reading the file as bytes, this code ran in .85 secs on my dual core 2.5Ghz PC targeting x86.

        Const one As Byte = 49 'ascii 1
        Dim cts As New List(Of Integer)

        Dim stpw As New Stopwatch
        Dim lg As Integer = animals(0).Length - 1
        stpw.Restart()
        For x As Integer = 0 To animals.Count - 2
            For y As Integer = 1 To animals.Count - 1
                Dim a1() As Byte = animals(x)
                Dim a2() As Byte = animals(y)
                Dim ct As Integer = 0
                For z As Integer = 0 To lg
                    If Not (a1(z) = one OrElse a2(z) = one) AndAlso a1(z) <> a2(z) Then
                        'If a1(z) <> a2(z) Then
                        ct += 1
                        'End If
                    End If
                Next
                cts.Add(ct)
            Next
        Next
        stpw.Stop()
        Label1.Text = stpw.Elapsed.ToString



Using VB 2012 Express I tried this targeting x64 and it was slower.

This post has been edited by dbasnett: 27 June 2013 - 07:25 AM

Was This Post Helpful? 0
  • +
  • -

#33 lar3ry  Icon User is offline

  • Coding Geezer
  • member icon

Reputation: 310
  • View blog
  • Posts: 1,290
  • Joined: 12-September 12

Re: Faster way to execute my loop

Posted 27 June 2013 - 02:45 PM

View PostC.Andrews, on 27 June 2013 - 07:39 AM, said:

The last 8 lines of the output file will have the run times for each segment in milliseconds. Currently the project is set to compile as a 64-bit application, but you can easily change that in the configuration manager.

I was curious as to the times per segment vs. total time, so I added another StopWatch as a Global, and got the following results:

String compare using 2 And 0
Segment 01 calculation time: 96
Segment 02 calculation time: 83
Segment 03 calculation time: 83
Segment 04 calculation time: 80
Segment 05 calculation time: 81
Segment 06 calculation time: 82
Segment 07 calculation time: 83
Segment 08 calculation time: 84
Total Time: 00:00:00.1625516

Xor on Bytes
Segment 01 calculation time: 111
Segment 02 calculation time: 86
Segment 03 calculation time: 85
Segment 04 calculation time: 88
Segment 05 calculation time: 84
Segment 06 calculation time: 85
Segment 07 calculation time: 85
Segment 08 calculation time: 84
Total Time: 00:00:00.1703025

As you can see, the total times are not a lot different between using a compare of a string character and an Xor on Bytes.

I did, however, manage to get a small improvement by using an OrElse instead of an Or. Here's the relevant code, including the Total Time Stopwatch (tottime). I got total times between about 110 msec and 153 msec.

        Dim AnimalCount As Integer = aryGenome.Count
        Dim GeneCount As Integer = aryGenome(0).Length
        tottime.Start()
        LoopTimer.Start()
        For i As Integer = 0 To AnimalCount - 2
            strCurrent = aryGenome(i)
            For j = (i + 1) To AnimalCount - 1
                strContrast = aryGenome(j)
                For k As Integer = 0 To GeneCount - 1
                    If (strCurrent(k) = "2"c And strContrast(k) = "0"c) OrElse (strCurrent(k) = "0"c And strContrast(k) = "2"c) Then intDiffCount += 1
                Next
                aryResults.Add(intDiffCount)
                intDiffCount = 0
            Next
        Next

        LoopTimer.Stop()
        tottime.Stop()
        Dim ts As TimeSpan = LoopTimer.Elapsed
        aryResults.Add(ts.Milliseconds)
        e.Result = aryResults

 ' and later...

            wrtOutput.WriteLine("Segment 08 calculation time: " & aryResult8(intLastLine).ToString)
            wrtOutput.WriteLine("Total Time: " & tottime.Elapsed.ToString)
            wrtOutput.Close()


The times, of course, vary a fair amount, probably depending on what else is running, but it would seem to me that you have about the best that can be done (someone please prove me wrong!).
Was This Post Helpful? 1
  • +
  • -

#34 dbasnett  Icon User is offline

  • D.I.C Addict
  • member icon

Reputation: 109
  • View blog
  • Posts: 603
  • Joined: 01-October 08

Re: Faster way to execute my loop

Posted 27 June 2013 - 02:58 PM

Is this for the same data file in post #6? What is the speed of your CPU? Your times, if the data file is the same, is significantly faster than mine.

Have you tried this targeting x86 and x64? I was a little surprised that x64 was slower.
Was This Post Helpful? 0
  • +
  • -

#35 C.Andrews  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 14
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Faster way to execute my loop

Posted 27 June 2013 - 03:02 PM

View Postdbasnett, on 27 June 2013 - 09:58 PM, said:

Is this for the same data file in post #6? What is the speed of your CPU? Your times, if the data file is the same, is significantly faster than mine.

Have you tried this targeting x86 and x64? I was a little surprised that x64 was slower.


Yeah, it's strange that the x64 build would run slower; I have to use a 64-bit build because I run out of memory processing the complete file when I do an x86 build, so I'm not really doing it for speed, but I'd still expect it to run at least AS fast if not faster.
Was This Post Helpful? 0
  • +
  • -

#36 dbasnett  Icon User is offline

  • D.I.C Addict
  • member icon

Reputation: 109
  • View blog
  • Posts: 603
  • Joined: 01-October 08

Re: Faster way to execute my loop

Posted 27 June 2013 - 04:10 PM

I may be doing something wrong that is causing the difference. Maybe larry can try both and see if he gets different results.
Was This Post Helpful? 0
  • +
  • -

#37 lar3ry  Icon User is offline

  • Coding Geezer
  • member icon

Reputation: 310
  • View blog
  • Posts: 1,290
  • Joined: 12-September 12

Re: Faster way to execute my loop

Posted 27 June 2013 - 04:36 PM

View Postdbasnett, on 27 June 2013 - 03:58 PM, said:

Is this for the same data file in post #6? What is the speed of your CPU? Your times, if the data file is the same, is significantly faster than mine.

Yes, the same data file. I have an AMD Athlon Quad Core running at 2.8 GHz, 4 GB DDR3 memory.
Were your times slower using the Project posted in #31?

Quote

Have you tried this targeting x86 and x64? I was a little surprised that x64 was slower.

I tried it, and x64 is a bit slower, with segment times just over 100 ms, and a total time of about 200 to 215 ms.
Was This Post Helpful? 0
  • +
  • -

#38 C.Andrews  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 14
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Faster way to execute my loop

Posted 28 June 2013 - 06:24 AM

Here's a thought on the x86 vs x64 performance differences: when I do my testing I always use a fully optimized build and I don't run it from inside the VS IDE. It makes quite a difference to run the executable independent of Visual studio, in my experience, so I ran a test with two "release" executables on the same system and got the following results:

64-bit Build:
Segment 01 calculation time: 158
Segment 02 calculation time: 156
Segment 03 calculation time: 111
Segment 04 calculation time: 109
Segment 05 calculation time: 107
Segment 06 calculation time: 111
Segment 07 calculation time: 110
Segment 08 calculation time: 110

32-bit Build:
Segment 01 calculation time: 205
Segment 02 calculation time: 203
Segment 03 calculation time: 117
Segment 04 calculation time: 114
Segment 05 calculation time: 119
Segment 06 calculation time: 115
Segment 07 calculation time: 118
Segment 08 calculation time: 117
Was This Post Helpful? 0
  • +
  • -

#39 lar3ry  Icon User is offline

  • Coding Geezer
  • member icon

Reputation: 310
  • View blog
  • Posts: 1,290
  • Joined: 12-September 12

Re: Faster way to execute my loop

Posted 28 June 2013 - 07:04 AM

View PostC.Andrews, on 28 June 2013 - 07:24 AM, said:

Here's a thought on the x86 vs x64 performance differences: when I do my testing I always use a fully optimized build and I don't run it from inside the VS IDE. It makes quite a difference to run the executable independent of Visual studio, in my experience, so I ran a test with two "release" executables on the same system and got the following results:


Wow! Just tried a few x64 tests, and I am completely amazed at the difference between running in the IDE and running standalone. Here's my results:

'x64 Debug version running in IDE
'	Segment 01 calculation time: 102
'	Segment 02 calculation time: 113
'	Segment 03 calculation time: 103
'	Segment 04 calculation time: 110
'	Segment 05 calculation time: 110
'	Segment 06 calculation time: 107
'	Segment 07 calculation time: 101
'	Segment 08 calculation time: 105
'	Total Time: 00:00:00.2137451
'	
'x64 Debug version running standalone
'	Segment 01 calculation time: 30
'	Segment 02 calculation time: 34
'	Segment 03 calculation time: 29
'	Segment 04 calculation time: 29
'	Segment 05 calculation time: 29
'	Segment 06 calculation time: 31
'	Segment 07 calculation time: 30
'	Segment 08 calculation time: 33
'	Total Time: 00:00:00.0612959
'
'x64 Release version running standalone
'	Segment 01 calculation time: 29
'	Segment 02 calculation time: 29
'	Segment 03 calculation time: 29
'	Segment 04 calculation time: 53
'	Segment 05 calculation time: 30
'	Segment 06 calculation time: 29
'	Segment 07 calculation time: 31
'	Segment 08 calculation time: 29
'	Total Time: 00:00:00.0589800


Was This Post Helpful? 0
  • +
  • -

#40 dbasnett  Icon User is offline

  • D.I.C Addict
  • member icon

Reputation: 109
  • View blog
  • Posts: 603
  • Joined: 01-October 08

Re: Faster way to execute my loop

Posted 28 June 2013 - 07:41 AM

When I ran my code as standalone I processed the entire file from post #6 in about .56 seconds, which is .3 faster than what I posted before. I am confused what the segment times are? Are you breaking this into pieces?

I noticed this

 Dim ts As TimeSpan = LoopTimer.Elapsed
 aryResults.Add(ts.Milliseconds)


If this is the time you are displaying then I think it is wrong. That is just the millisecond component of the timespan. I think you want the .TotalMilliseconds

This post has been edited by dbasnett: 28 June 2013 - 07:54 AM

Was This Post Helpful? 1
  • +
  • -

#41 C.Andrews  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 14
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Faster way to execute my loop

Posted 28 June 2013 - 08:13 AM

View Postdbasnett, on 28 June 2013 - 02:41 PM, said:

When I ran my code as standalone I processed the entire file from post #6 in about .56 seconds, which is .3 faster than what I posted before. I am confused what the segment times are? Are you breaking this into pieces?

I noticed this

 Dim ts As TimeSpan = LoopTimer.Elapsed
 aryResults.Add(ts.Milliseconds)


If this is the time you are displaying then I think it is wrong. That is just the millisecond component of the timespan. I think you want the .TotalMilliseconds


The project I posted yesterday does break the data up into 8 pieces for parallel processing, using a number of background workers; that's why you're seeing times for each segment.

Man, I'm glad you pointed out my error with the ts.Milliseconds. I didn't know I needed .TotalMilliseconds. It doesn't matter for our testing because we're never going over 1,000 MS with the small datafile, but it takes a whole lot longer for the large datafiles. Good catch!
Was This Post Helpful? 0
  • +
  • -

#42 dbasnett  Icon User is offline

  • D.I.C Addict
  • member icon

Reputation: 109
  • View blog
  • Posts: 603
  • Joined: 01-October 08

Re: Faster way to execute my loop

Posted 28 June 2013 - 08:31 AM

From what I can tell the code I posted and ran from x86 was almost twice as fast, assuming I can add all the segment times correctly, as your x64. And larry's code times were just amazingly faster.

At this point have we made progress towards speeding up the processing of the larger data set?
Was This Post Helpful? 0
  • +
  • -

#43 C.Andrews  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 14
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Faster way to execute my loop

Posted 28 June 2013 - 08:42 AM

View Postdbasnett, on 28 June 2013 - 03:31 PM, said:

From what I can tell the code I posted and ran from x86 was almost twice as fast, assuming I can add all the segment times correctly, as your x64. And larry's code times were just amazingly faster.

At this point have we made progress towards speeding up the processing of the larger data set?


From where I started with my original post, we've made significant performance enhancements. My total processing time has been decreased by about 70% using optimizations suggested by previous posters. We have made minimal, but still notable improvements since yesterday by tweaking my conditional statement into a more optimal form.
Was This Post Helpful? 0
  • +
  • -

#44 dbasnett  Icon User is offline

  • D.I.C Addict
  • member icon

Reputation: 109
  • View blog
  • Posts: 603
  • Joined: 01-October 08

Re: Faster way to execute my loop

Posted 28 June 2013 - 09:26 AM

One other thought. Instead of breaking this up into some arbitrary number of background workers I would break it up into a background worker / CPU.
Was This Post Helpful? 0
  • +
  • -

#45 C.Andrews  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 14
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Faster way to execute my loop

Posted 28 June 2013 - 09:38 AM

View Postdbasnett, on 28 June 2013 - 04:26 PM, said:

One other thought. Instead of breaking this up into some arbitrary number of background workers I would break it up into a background worker / CPU.


That's what I'm doing, the computer on which this will be run by the end user is an 8-core machine.
Was This Post Helpful? 0
  • +
  • -

  • (4 Pages)
  • +
  • 1
  • 2
  • 3
  • 4