Memory usage problem

  • (3 Pages)
  • +
  • 1
  • 2
  • 3

38 Replies - 4806 Views - Last Post: 21 November 2012 - 04:00 PM Rate Topic: -----

#1 C.Andrews   User is offline

  • D.I.C Head
  • member icon

Reputation: 15
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Memory usage problem

Posted 21 November 2012 - 08:09 AM

In my application, it becomes necessary to launch a 3rd party program that processes a large amount of data that have been prepared by my program. The program I'm using to process the data is executed through the command prompt, and my program writes an MSDOS batch file which submits the jobs according to which data are available, and the processing takes place on a linux server.

Now, once the processing is complete, my program performs clean-up and moves some of the data around, so I need my application to idle for an indeterminate period of time (from 10 mins to several hours) while it waits for the external program to finish. To accomplish this, I'm using a function that checks if the cmd.exe process is running, and then waits for a 1/2 second (using Threading.Thread.Sleep(500)) before checking again. Once it sees that cmd.exe is no longer running, it performs the clean-up operations and all is well.

The problem I'm having is that while my program is idling, it consumes ever-increasing amounts of memory. If the jobs I submit take an exceptional amount of time to run, the program runs itself out of memory and explodes. I can't figure out why calling my function and putting the thread to sleep every 1/2 second is using so much memory.

Anyone who has read one of my posts is probably aware by now that I am NOT a programmer, just a dude who likes to splash around in water way over his head, so if you think you have a better solution to my idling problem, feel free to offer it in the same manner you'd offer advice to a small, dull child. Thanks!

Here's the function that checks for CMD

    Public Function CheckForCMD(ByVal name As String) As Boolean
        For Each clsProcess As Process In Process.GetProcesses()
            If clsProcess.ProcessName.Contains(name) Then
                Return True
            End If
        Next
        Return False
    End Function



Here's the loop that calls the function:

        Do While CheckForCMD("cmd") = True
            Threading.Thread.Sleep(500)
        Loop



Is This A Good Question/Topic? 0
  • +

Replies To: Memory usage problem

#2 modi123_1   User is online

  • Suitor #2
  • member icon



Reputation: 14095
  • View blog
  • Posts: 56,486
  • Joined: 12-June 08

Re: Memory usage problem

Posted 21 November 2012 - 08:16 AM

That small amount of code isn't really telling a large picture of what is going on. Are all the right objects disposed? Is the check calling itself again? Is there some sort of loop causing havoc? Then there's questions of - why is this running every half second? Is it that crucial this data be cleared *RIGHT NOW*? If not it might be worth splitting the clean up part into a separate app that's first command (when ran) is to check for your cmd.exe condition. If it's running then shut down... then have window's task scheduler run this clean up app say every 15 minutes. No muss, no fuss sort of thing.
Was This Post Helpful? 1
  • +
  • -

#3 tlhIn`toq   User is offline

  • Xamarin Cert. Dev.
  • member icon

Reputation: 6534
  • View blog
  • Posts: 14,450
  • Joined: 02-June 10

Re: Memory usage problem

Posted 21 November 2012 - 08:19 AM

My first thought is as I often say "thread.sleep" is EVIL.

Everything on the thread sleeps. Everything. It is not
thread.WaitButContinueToPerformBackgroundOperations(500)
which might include things like Garbage Collection.

Every half second you create a new collection from Process.GetProcesses() and I'm going to guess that with your thread spending most of its life asleep it never gets a chance to garbage collect.

Try inserting GC.Collect() between lines 7 & 8 so it clears just before exiting the routine.

Second: If this process can take a minimum of 10 minutes do you really need to hammmer the hell out of this every 1/2 second? Wouldn't ever 5-10 seconds be fine?

This post has been edited by tlhIn`toq: 21 November 2012 - 08:21 AM

Was This Post Helpful? 1
  • +
  • -

#4 C.Andrews   User is offline

  • D.I.C Head
  • member icon

Reputation: 15
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Memory usage problem

Posted 21 November 2012 - 08:56 AM

View PosttlhIn`toq, on 21 November 2012 - 08:19 AM, said:

Try inserting GC.Collect() between lines 7 & 8 so it clears just before exiting the routine.


This didn't work.

Quote

Second: If this process can take a minimum of 10 minutes do you really need to hammmer the hell out of this every 1/2 second? Wouldn't even 5-10 seconds be fine?


The problem is that I don't want the user to have to wait after the processing step is complete to be able to use the application; if I increase the delay between checks, the time for the application to be available again after processing would be noticeable depending on where the loop is when cmd.exe closes.

The clean-up process has to happen immediately, because part of that subroutine unzips files generated during the processing and makes them available for the next step and oh hell that's the easy answer...

The user will always immediately proceed to the next step (which is clicking another button) once the process is finished, so I just need to move my clean-up post-processing to the start of the next step instead of the end of this step. God I'm retarded sometimes.
Was This Post Helpful? 0
  • +
  • -

#5 tlhIn`toq   User is offline

  • Xamarin Cert. Dev.
  • member icon

Reputation: 6534
  • View blog
  • Posts: 14,450
  • Joined: 02-June 10

Re: Memory usage problem

Posted 21 November 2012 - 09:31 AM

View PostC.Andrews, on 21 November 2012 - 09:56 AM, said:

View PosttlhIn`toq, on 21 November 2012 - 08:19 AM, said:

Try inserting GC.Collect() between lines 7 & 8 so it clears just before exiting the routine.


This didn't work.

Quote

Second: If this process can take a minimum of 10 minutes do you really need to hammmer the hell out of this every 1/2 second? Wouldn't even 5-10 seconds be fine?


The problem is that I don't want the user to have to wait after the processing step is complete to be able to use the application; if I increase the delay between checks, the time for the application to be available again after processing would be noticeable depending on where the loop is when cmd.exe closes.


Again, THREAD.SLEEP is evil. That's why your application is not available and seems locked up: Because you have everything happening on one thread including the GUI and you are sleeping that. You could/should put your timer/cmd.exe check on a different thread so it doesn't tie up the GUI. And don't use thread.sleep. Throw in a timer for 10 seconds, then on the Timer.Tick have your program do the check.

Quote

The clean-up process has to happen immediately, because part of that subroutine unzips files generated during the processing and makes them available for the next step and oh hell that's the easy answer...

You don't know how many times the answer comes to all of us as soon as we start having to describe it to someone else. People think I'm nuts talking to my poodle about coding (I work at home), but the act of talking it through OUT LOUD does wonders. So many problems suddenly become clear when we hear them.
Was This Post Helpful? 2
  • +
  • -

#6 C.Andrews   User is offline

  • D.I.C Head
  • member icon

Reputation: 15
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Memory usage problem

Posted 21 November 2012 - 09:55 AM

[/quote]

You could/should put your timer/cmd.exe check on a different thread so it doesn't tie up the GUI. And don't use thread.sleep. Throw in a timer for 10 seconds, then on the Timer.Tick have your program do the check.

[quote]

I thought this same thing, but I didn't do it because it wouldn't resolve the underlying issue: that of out-of-control memory usage. If I launch my check-for-cmd subroutine in another thread, I can indeed avoid locking my main gui (I actually do this a number of times for background processes elsewhere in the application); however, without a different method for checking if cmd.exe is finished, the new thread would just grow ponderous in memory and explode instead of the main ui exploding.

Out of curiosity, I'll try using a timer instead of thread.sleep and see if the memory problem is resolved, even though I've circumvented the problem by moving the cleanup process to another step. Never know when I might need to stall a thread for a bit in another application.
Was This Post Helpful? 0
  • +
  • -

#7 tlhIn`toq   User is offline

  • Xamarin Cert. Dev.
  • member icon

Reputation: 6534
  • View blog
  • Posts: 14,450
  • Joined: 02-June 10

Re: Memory usage problem

Posted 21 November 2012 - 10:06 AM

If you haven't done it, then how do you know it wouldn't resolve the issue? Off loading the working that creates the memory leak to another thread that doesn't sleep may or may not resolve the issue. But you won't know until you try. And trying this simple idea would have taken less time than we've spent talking about it.

Quote

however, without a different method for checking if cmd.exe is finished, the new thread would just grow ponderous in memory and explode instead of the main ui exploding.
All part of the diagnostic process. If you can watch that specific thread grow then you have a limited amount of code that you have CONFIRMED is the culprit.
Was This Post Helpful? 0
  • +
  • -

#8 C.Andrews   User is offline

  • D.I.C Head
  • member icon

Reputation: 15
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Memory usage problem

Posted 21 November 2012 - 12:04 PM

View PosttlhIn`toq, on 21 November 2012 - 10:06 AM, said:

If you haven't done it, then how do you know it wouldn't resolve the issue? Off loading the working that creates the memory leak to another thread that doesn't sleep may or may not resolve the issue. But you won't know until you try. And trying this simple idea would have taken less time than we've spent talking about it.

Quote

however, without a different method for checking if cmd.exe is finished, the new thread would just grow ponderous in memory and explode instead of the main ui exploding.
All part of the diagnostic process. If you can watch that specific thread grow then you have a limited amount of code that you have CONFIRMED is the culprit.


So, I wrote a completely new program just to mess around with this problem. Here it is:

Imports Scripting
Public Class Form1

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Timer1.Start()
        Dim oOutFileO As New FileSystemObject
        Dim oOutFile = oOutFileO.CreateTextFile("doesnothing.bat", True)
        oOutFile.Writeline("pause")
        oOutFile.close()

        System.Diagnostics.Process.Start("doesnothing.bat")

      End Sub
    Public Function CheckForCMD(ByVal name As String) As Boolean
        For Each clsProcess As Process In Process.GetProcesses()
            If clsProcess.ProcessName.Contains(name) Then
                Return True
            End If
        Next
        Return False
    End Function

    Private Sub Timer1_Tick(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Timer1.Tick
        CheckForCMD("cmd")
    End Sub
End Class



Using the timer doesn't make any difference, it still accumulates memory with each tick event. Clearly the problem is the function I'm using to check for cmd.exe. It must be storing the process information in memory over and over and not getting rid of it, but I don't know how to tell it to purge those data each time it checks.

Also, I retooled my mini-application to run the function in a second thread using both System.Threading and the Background Worker control, but as I hypothesized, the the application still sucks down memory.
Was This Post Helpful? 0
  • +
  • -

#9 lucky3   User is offline

  • Friend lucky3 As IHelpable
  • member icon

Reputation: 233
  • View blog
  • Posts: 770
  • Joined: 19-October 11

Re: Memory usage problem

Posted 21 November 2012 - 12:12 PM

Hah... I just wrote about disposing. You might find it useful for your problem.
Was This Post Helpful? 0
  • +
  • -

#10 tlhIn`toq   User is offline

  • Xamarin Cert. Dev.
  • member icon

Reputation: 6534
  • View blog
  • Posts: 14,450
  • Joined: 02-June 10

Re: Memory usage problem

Posted 21 November 2012 - 12:20 PM

Looks like a really good and helpful tutorial.
VB.NET is not my normal language, but looking at this I don't see an actual object the OP *could* dispose of. They aren't actually making an object of any kind. The foreach loop should be temporarily pulling a collection from Process.GetProcesses() but once the method is returned that temporary collection should be marked as available for GC.

At this point I'm with the OP - I'm not sure where the memory is being consumed, especially since they forced collection in an earlier test.


14    Public Function CheckForCMD(ByVal name As String) As Boolean
15        For Each clsProcess As Process In Process.GetProcesses()
16            If clsProcess.ProcessName.Contains(name) Then
17                Return True
18            End If
19        Next
20        Return False
21    End Function


Was This Post Helpful? 0
  • +
  • -

#11 C.Andrews   User is offline

  • D.I.C Head
  • member icon

Reputation: 15
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Memory usage problem

Posted 21 November 2012 - 12:26 PM

View Postlucky3, on 21 November 2012 - 12:12 PM, said:

Hah... I just wrote about disposing. You might find it useful for your problem.


You da man, Lucky. This totally works.
Was This Post Helpful? 0
  • +
  • -

#12 it.director   User is offline

  • New D.I.C Head

Reputation: 3
  • View blog
  • Posts: 20
  • Joined: 20-November 12

Re: Memory usage problem

Posted 21 November 2012 - 12:27 PM

Your process starts an external app
run that app manually as your code would and see if its own process increases memory. May be out of your control.

also... try
If Process.GetProcessesByName(name).Length > 0 Then ' Do Stuff ...


Was This Post Helpful? 0
  • +
  • -

#13 C.Andrews   User is offline

  • D.I.C Head
  • member icon

Reputation: 15
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Memory usage problem

Posted 21 November 2012 - 12:37 PM

View Postit.director, on 21 November 2012 - 12:27 PM, said:

Your process starts an external app
run that app manually as your code would and see if its own process increases memory. May be out of your control.

also... try
If Process.GetProcessesByName(name).Length > 0 Then ' Do Stuff ...



The app that my program launches is just an ms dos batch file that I construct on the fly which submits a bunch of DMU jobs to another server. I can launch it independently of my application and it works just fine without ever increasing memory problems, the actual work doesn't even take place on the same machine as my application is running.
Was This Post Helpful? 0
  • +
  • -

#14 lucky3   User is offline

  • Friend lucky3 As IHelpable
  • member icon

Reputation: 233
  • View blog
  • Posts: 770
  • Joined: 19-October 11

Re: Memory usage problem

Posted 21 November 2012 - 12:40 PM

View PosttlhIn`toq, on 21 November 2012 - 08:20 PM, said:

VB.NET is not my normal language, but looking at this I don't see an actual object the OP *could* dispose of. They aren't actually making an object of any kind. The foreach loop should be temporarily pulling a collection from Process.GetProcesses() but once the method is returned that temporary collection should be marked as available for GC.


Yeah tlhIn`toq, I agree with all that. But what else could consume resources? Process class is having Handle property, then GetProcesses method creates array of Process class components, each with their own handle...

Edit:
here we go:

Quote

the actual work doesn't even take place on the same machine as my application is running.

This post has been edited by lucky3: 21 November 2012 - 12:42 PM

Was This Post Helpful? 0
  • +
  • -

#15 C.Andrews   User is offline

  • D.I.C Head
  • member icon

Reputation: 15
  • View blog
  • Posts: 169
  • Joined: 18-October 12

Re: Memory usage problem

Posted 21 November 2012 - 12:48 PM

View PostC.Andrews, on 21 November 2012 - 12:26 PM, said:

View Postlucky3, on 21 November 2012 - 12:12 PM, said:

Hah... I just wrote about disposing. You might find it useful for your problem.


You da man, Lucky. This totally works.


Hmm, I may have spoken too quickly. For whatever reason, the dispose method only fixes the memory leak inside the visual basic environment. If I compile the program and run it independently, it still sucks down memory. Most likely I just don't have any idea what I'm doing.
Was This Post Helpful? 0
  • +
  • -

  • (3 Pages)
  • +
  • 1
  • 2
  • 3