12 Replies - 854 Views - Last Post: 20 July 2011 - 01:06 PM Rate Topic: -----

#1 theseamusjames  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 27
  • Joined: 17-May 10

Generating reports, timing out

Posted 18 July 2011 - 04:07 PM

I'm trying to generate a very very large report for my site, but I'm having problems with it timing out. I'm currently using

set_time_limit (0);


but it's still timing out. I'd say it's probably in the neighborhood of 2min before it times out. I also tried having it send an email at the end of the function to see if the process was maybe continuing on the server but the browser was timing out, but that didn't seem to work either.

Basically, we're trying to generate a report about every single transaction we've ever done (which is in the 10s of thousands). Any ideas?

Is This A Good Question/Topic? 0
  • +

Replies To: Generating reports, timing out

#2 CTphpnwb  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 3001
  • View blog
  • Posts: 10,393
  • Joined: 08-August 08

Re: Generating reports, timing out

Posted 18 July 2011 - 04:29 PM

Do them in batches, and append the results to a file. That way you can output a few hundred, verify that they're good, back up the file and run it again. Repeat as necessary.
Was This Post Helpful? 1
  • +
  • -

#3 theseamusjames  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 27
  • Joined: 17-May 10

Re: Generating reports, timing out

Posted 18 July 2011 - 04:34 PM

Good call, thanks. I can probably make that work.
Was This Post Helpful? 0
  • +
  • -

#4 satis  Icon User is offline

  • D.I.C Head

Reputation: 82
  • View blog
  • Posts: 231
  • Joined: 26-May 11

Re: Generating reports, timing out

Posted 19 July 2011 - 09:39 AM

The execution time limit could be on the host as well. If you're on a shared host, that wouldn't be too surprising. It may be a good idea to just grab your DB and do processing locally, instead. Load it up in a local box where you have full control of PHP, or even write a separate application in a different language to do the processing. Something like C++ or C# could probably process it faster than PHP could, assuming you can write in those languages. If you are on a shared host, they won't come down on you for excessive CPU and/or memory utilization, too.
Was This Post Helpful? 0
  • +
  • -

#5 theseamusjames  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 27
  • Joined: 17-May 10

Re: Generating reports, timing out

Posted 19 July 2011 - 02:10 PM

Actually, dividing them in to blocks won't really work either. We're grouping these transactions based on what they apply to, so if something included like 10,000 transactions, it would timeout.

We're not on a shared host, so we can alter our configuration. But conceivably, just setting that max execution time should do it, right? I tried set_time_limit and ini_set to no avail. Ideas?
Was This Post Helpful? 0
  • +
  • -

#6 satis  Icon User is offline

  • D.I.C Head

Reputation: 82
  • View blog
  • Posts: 231
  • Joined: 26-May 11

Re: Generating reports, timing out

Posted 19 July 2011 - 05:27 PM

I'd definitely check php.ini max_execution_time. It could also be memory_limit. It's possible loading all this up is running into that. If that doesn't help, maybe crank up your error_reporting to E_ALL and make sure display_errors is on. Perhaps that'll give you some idea. Also check php or apache error logs in case they contain something of interest.

Actually, you mention in your first post that it's timing out. Do you have anything indicating that specifically, or is it just dying after however long and you're assuming it's a timeout problem? Any specific errors you're receiving may help.
Was This Post Helpful? 0
  • +
  • -

#7 CTphpnwb  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 3001
  • View blog
  • Posts: 10,393
  • Joined: 08-August 08

Re: Generating reports, timing out

Posted 19 July 2011 - 05:31 PM

Without seeing code we're just guessing. It could be lots of things.
Was This Post Helpful? 0
  • +
  • -

#8 theseamusjames  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 27
  • Joined: 17-May 10

Re: Generating reports, timing out

Posted 20 July 2011 - 11:24 AM

View Postsatis, on 19 July 2011 - 05:27 PM, said:

I'd definitely check php.ini max_execution_time. It could also be memory_limit. It's possible loading all this up is running into that. If that doesn't help, maybe crank up your error_reporting to E_ALL and make sure display_errors is on. Perhaps that'll give you some idea. Also check php or apache error logs in case they contain something of interest.


Max execution time in php.ini is set to 4000. Memory limit is 128mb. Error reporting is on (and I receive an email error report every time one occurs). No errors being displayed. Httpd.conf timeout is set to 1200 (shouldn't take longer than 10min, from what I've seen).

Also, the script works on my local box (where all the settings are the same).

View Postsatis, on 19 July 2011 - 05:27 PM, said:

Actually, you mention in your first post that it's timing out. Do you have anything indicating that specifically, or is it just dying after however long and you're assuming it's a timeout problem? Any specific errors you're receiving may help.


It appears to be a timeout problem because it spins and spins and spins, and then it stops spinning and nothing is on the screen (nor in the source of the page). I also put in some echos for the sake of debugging (in the loops and before the loops), which never make it to the screen. I also tried sending headers at the very start of the function (for outputting a CSV file), also to no avail.

Sorry about not sharing code, but it's small module of a much larger MVC application and without context it won't be clear. However, I'm 1000% sure it's a timeout issue related to execution time or the browser timing out. When my local settings weren't set to long enough, it had the same behavior. When I fixed them, it began working.

Eventually, the plan will be for a user to request the report, it will generate via AJAX, and email them when the report is finished.
Was This Post Helpful? 0
  • +
  • -

#9 CTphpnwb  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 3001
  • View blog
  • Posts: 10,393
  • Joined: 08-August 08

Re: Generating reports, timing out

Posted 20 July 2011 - 11:30 AM

Well then start looking for infinite loops.
Was This Post Helpful? 0
  • +
  • -

#10 theseamusjames  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 27
  • Joined: 17-May 10

Re: Generating reports, timing out

Posted 20 July 2011 - 12:00 PM

View PostCTphpnwb, on 20 July 2011 - 11:30 AM, said:

Well then start looking for infinite loops.


I don't think you understand. The same code that works flawlessly on our development server - the exact same code - does not work on our production server.

The issue is with set_time_limit (0) and ini_set ( 'max_execution_time', 100000 ) not working correctly. We're still experiencing time limit issues despite using these functions to override the server settings, which are also configured to allow really long executions.

Having a working copy on our dev server is great, and whenever we want this report we can always mirror our production DB locally and then run the report, but that's not going to cut it in the long run.
Was This Post Helpful? 0
  • +
  • -

#11 CTphpnwb  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 3001
  • View blog
  • Posts: 10,393
  • Joined: 08-August 08

Re: Generating reports, timing out

Posted 20 July 2011 - 12:12 PM

Well then it's likely that your host is blocking something you're doing.
Was This Post Helpful? 0
  • +
  • -

#12 satis  Icon User is offline

  • D.I.C Head

Reputation: 82
  • View blog
  • Posts: 231
  • Joined: 26-May 11

Re: Generating reports, timing out

Posted 20 July 2011 - 12:47 PM

There are a few other things you could try. fopen a file and write status updates to it. Even if echoes don't show up because the script is being terminated before shunting its results to the web server, a file is written in real time. That could give you something.

Do you have the ability to execute code via a system call? Something along the lines of

exec("/usr/local/bin/php myscript.php >> output.log &");


This starts a secondary process, separate from the one rendering the page making that call, and puts it in the background. It won't (can't) render anything to the browser, but it can execute and anything it outputs is redirected to output.log (in my example).

I'm not sure if either of those will tell you anything more, but I don't have many more ideas. The fork should tell you if it's an apache issue, since it bypasses the webserver and just directly invokes the PHP parser. Either could be used to verify you're not running into some sort of infinite loop or error condition that is only happening on the production server for some reason. Unlikely, sure, but weird things happen all the time.

This post has been edited by satis: 20 July 2011 - 12:47 PM

Was This Post Helpful? 0
  • +
  • -

#13 theseamusjames  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 27
  • Joined: 17-May 10

Re: Generating reports, timing out

Posted 20 July 2011 - 01:06 PM

View Postsatis, on 20 July 2011 - 12:47 PM, said:

Do you have the ability to execute code via a system call? Something along the lines of

exec("/usr/local/bin/php myscript.php >> output.log &");


This starts a secondary process, separate from the one rendering the page making that call, and puts it in the background.


This is beautiful. Only drawback is our framework (we're on codeigniter) means that we can't run the script directly, so either I have to rewrite the script without all the dependencies (not a good idea - huge amounts of databse queries, object construction, etc), or figure out a way to exec an mvc function.

We run our own ec2 server, so we can do pretty much anything we want. This is a great way for us to create these reports independently of the user's actions. I was going to use AJAX for that same reason - allowing me to run a second instance of something in the background, but it's still subject to the same apache constraints.

Thanks.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1