[FredrikDuwell]

Hey everyone!
This is my first topic here, and I'm going to introduce myself really quick.
My name is Fredrik Duwell (obviously) and I come from the southern part of Sweden. I'm a PHP-programmer, mostly dedicated on my sparetime etc.

Today I've got a problem with one of my new projects. It's a service we were supposed to publish a few days ago, but we're delayed just because of this small issue. I bet it is kind of easy to solve, but I'm out of ideas.

Here we go!

When a file is being uploaded, there's no problem. There's however an issue when the file is being downloaded because Apache freezes until the file is completely downloaded, and when I say "freezes" I really mean for everyone. None can access the website until download is completed.

I'm using a method that reads the file in some way, sends it in pieces etc. Here, have a look at the source;

$file['location'] = @fopen($config['filetransfer']['directory'].'/'.$download['data']['fetch']['id_unique'].'.'.$download['data']['fetch']['filename_extension']);
$file['filesize'] = @filesize($config['filetransfer']['directory'].'/'.$download['data']['fetch']['id_unique'].'.'.$download['data']['fetch']['filename_extension']);

if($file['location']) {
	// Create the headers used for downloading the file
	header("Content-Transfer-Encoding: binary");
	header("Expires: 0");
	header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
	header("Content-Type: application/force-download");
	header("Accept-Ranges: bytes");
	header("Content-Disposition: attachment; filename='{$config['filetransfer']['prefix']}{$download['data']['fetch']['filename']}");
	header("Content-Length: {$file['filesize']}");
	// Actually start downloading the file
	
	while (!feof($file['location'])) { 
	   echo(@fgets($file['location'], 4096)); 
	}
	
	/**
	 * Update the total of
	 * downloaded in user's account.
	 */
         /* Removed, hehe */

	/**
	 * Close the stream
	 */
	fclose($file['location']);
}

I've also searched for every solution, tried a few but nothing seems to work. I don't feel very lucky right now!
I'd be more than happy if someone took the initiative and helped me to solve this.

Thank you.

[JackOfAllTrades]

Have you tried fpassthru by any chance?

There are so many variables here (pardon the pun). Is this a huge file? Have you monitored memory usage on the system at this time? What kind of load is on the system at the time that it "freezes"? Is output buffering on?

[CTphpnwb]

Just a guess, but you may have a problem with your cache or expire headers.
Try it with no expire header and:

header("Cache-Control: public");

[FredrikDuwell]

JackOfAllTrades, on 10 June 2012 - 04:46 PM, said:

Have you tried fpassthru by any chance?

There are so many variables here (pardon the pun). Is this a huge file? Have you monitored memory usage on the system at this time? What kind of load is on the system at the time that it "freezes"? Is output buffering on?

I tried it and it works, but the download doesn't start directly. For a big file around 1-1.5GB it takes around 6-7 seconds before the download actually starts. It isn't supposed to do that. It works great otherwise!

CTphpnwb, on 10 June 2012 - 08:25 PM, said:

Just a guess, but you may have a problem with your cache or expire headers.
Try it with no expire header and:

header("Cache-Control: public");

I did this first and as expected there was no change. I don't really think those headers would have any affect. Sorry!

In short, the problem still not solved. I have continued to search through half the net to find a solution, but only seems to be only me who has this problem. Before we connected to FTP servers to download the files, but even there everything stopped working right and website was inaccessible, at least until the file was downloaded or interrupted.

[Atli]

Try using the readfile() function to send the file, rather than using fopen() and sending it manually.

[FredrikDuwell]

I've tried that too, but unfortunately that doesn't hide the "real path", if you know what I mean. I wouldn't like people (or bots) to bypass the captcha, or even worse, have direct download from our servers.

[JackOfAllTrades]

FredrikDuwell, on 11 June 2012 - 05:39 AM, said:

JackOfAllTrades, on 10 June 2012 - 04:46 PM, said:

Have you tried fpassthru by any chance?

There are so many variables here (pardon the pun). Is this a huge file? Have you monitored memory usage on the system at this time? What kind of load is on the system at the time that it "freezes"? Is output buffering on?

I tried it and it works, but the download doesn't start directly. For a big file around 1-1.5GB it takes around 6-7 seconds before the download actually starts. It isn't supposed to do that. It works great otherwise!

Not positive, but this sounds to me like PHP is configured with output buffering on, which may be causing memory to get filled up prior to starting the sending process, especially if this is combined with high memory limits.

[FredrikDuwell]

Ok, so a possible solution would be to lower the memory limit?

[Atli]

FredrikDuwell, on 11 June 2012 - 10:25 AM, said:

I've tried that too, but unfortunately that doesn't hide the "real path", if you know what I mean. I wouldn't like people (or bots) to bypass the captcha, or even worse, have direct download from our servers.

I think you are misunderstanding what that function does. It doesn't show the user the "real path", it just pushes the file data into the output buffer. It's basically the exact some as doing this, except bypassing the whole "fopen" and "fclose" part.

$fh = fopen("/path/to/file", "rb");
fpassthru($fh);
fclose($fh);

The fopen() family of functions are more aimed at allow the code to interact with the file data, rather than just dumping it into the output buffer, which is all you really need and is what readfile() does.

Quote

Ok, so a possible solution would be to lower the memory limit?

I doubt it. The first thing I'd do is see if output buffering or output compression is on. This should help with that:

<?php
header('Content-Type: text/plain; charset=UTF-8');
echo "Buffering: " . (ini_get("output_buffering") ?: "Off") . PHP_EOL;
echo "Compression: " .(ini_get("zlib.output_compression") ?: "Off")  . PHP_EOL;

If they are on, you could turn them off using a .htaccess file. Just adding these two lines should do it:

php_flag output_buffering 0
php_flag zlib.output_compression 0

The zlib.output_compression flag can also be set in the top of your PHP scripts with ini_set, but the output_buffering flag must be set before the script is executed. (Ideally in the main php.ini file, but .htaccess works too.)

[FredrikDuwell]

I did understand that, and both compression and buffer was turned off. When I was looking at the memory usage while downloading a file of 1.5GB it went from 2.3GB straight up to 4GB in 0.3 seconds. Is this even possible?

It seems like my way of programming doesn't work at all, I'll have to hire someone. Damn it!

[Atli]

OK, that definitely doesn't sound good. You need a way to allow Apache itself to send the file, rather than have PHP read the file into the output buffer.

After some research I found the mod_xsendfile Apache module. It'll let you send a file path from PHP to Apache as a HTTP header, which Apache will then take and send to the browser in small chunks, so that it will never have to load more than a fraction of it into memory at once. That should greatly improve your memory usage.

I actually just tried this myself on one of my test servers, and on a server with 512MB RAM total I was able to send a 1GB file from PHP without any noticeable memory increase in Apache, and without any drop in download speeds.


This is how you do it. Assuming you've already installed the module -- which is very simple; there are instructions on the page I linked to -- you need to update your Apache config to include two new directives:

XSendFile On
XSendFilePath /path/to/file/dir

The XSendFilePath value should be the path to the local file directory, where the files are actually stored. It adds it to the white-list of directories Apache is allowed to send files from. (The module will hide the local path from the browser.)

Once that is ready, you'll have to change your PHP code to set the X-Sendfile header to the local path of the file you want to send. You may also want to set the Content-Disposition header, if you want to force download and set a name for the file. Other headers, such as caching headers or Content-Length, should be skipped. The module will take care of that for you.

So that would look something like:

$filePath = "/path/to/file/dir/thefile.ext";
header("Content-Disposition: attachment; filename='" . basename($filePath) . "'");
header('X-Sendfile: ' . $filePath);
exit;

And that's it. Apache will now send the file to the browser in as an efficient manner as possible.


There is one precaution you may want to take though, in case the Apache server is reconfigured, or the script is moved or something, and the mod_xsendfile module becomes unavailable. You can use the apache_get_modules() function to check if it is indeed loaded, so you aren't accidentally sending the local file path to the browser. (Apache won't stop you from sending the X-Sendfile header directly to the client if the module isn't there to intercept it.)

if (!in_array("mod_xsendfile", apache_get_modules())) {
    die("Server configuration error: XSendFile module not available!");
}

(Note that this won't detect if the XSendFile directive is Off!)

[mikeneiderhauser]

Is there any other way around this issue? I need to keep these files secure, and my web host will not let me install the mod_xsendfile as mentioned in previous posts. I am working with large files when trying to modify the header to download and I cannot find a solution to the issue at hand.

Thanks!

[mikeneiderhauser]

Just to give a little more info on what I am doing.....

I am storing these "large files" (~1GB or greater) outside of the public_html folder. I am currently able to download them using the open and send Xbyte of the file at a time. However.. when this function is called, I can not make any other requests in that browser. I click on a second download link that uses the same function (with a different filepath) to download another large file while the other large file is being handled.. The webpage freezes until the first download is finished and then starts the next download. This is not a huge issue, however... this also occurs when downloading a "large file" and then trying to load a different page on the site.

I have also tried doing the fpasstrhough. This downloads the file, but without any content.

Any suggestions that could help with this issue? Below is the code that is called by a html link and it opens a new tab which (after user verification) runs said code. (get_mime() will return the proper mime type for the file being downloaded)

Thanks

function send_download($file_path)
{
	//get infor of file
	$path_parts=pathinfo($file_path);
	$filename = $path_parts['basename'];
	//extract extension
	$file_part_ext = $path_parts['extension'];
	//get file size
	$file_size=@filesize($file_path);
	
	//determine Content-Type attrib for file download
	$mime_type = get_mime(".".$file_part_ext);
	
	//process header
	header("Cache-Control: public");
	header("Content-Type: $mime_type");
	header("Content-disposition: attachment; filename=$filename");
	header("Content-Length: ".($file_size));
	set_time_limit(0); 
	readfile_chunked($file_path);
	exit(0);
}//end send_download


function readfile_chunked ($filename) { 
	set_time_limit(0); 
	$blocksize = (2 << 20); //2M chunks
	$file = @fopen($filename,"rb");
		while(!feof($file))
		{
			print(@fread($file, $blocksize));
			ob_flush();
			flush();
		}
	return fclose($file);
}