File upload doesnt caching...

  • (3 Pages)
  • +
  • 1
  • 2
  • 3

30 Replies - 2760 Views - Last Post: Yesterday, 11:29 AM

#1 Radius Nightly  Icon User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 50
  • Joined: 07-May 15

File upload doesnt caching...

Posted 02 October 2017 - 05:56 AM

Didnt come for a long time here. Hi to all.

I set up older server on Windows PoS, with Nginx v1.13.4, FastCGI v5.4.9 and MySQL v5.7.1 m11. And i got now problems with FastCGI (i dont think its about Nginx).

Its about file upload. Everything worked fine, until i bring up FastCGI settings (it was without settings for a long time). Downloading are still fine from both sides (Nginx Caching and FastCGI), but uploading getting problems (for users and for server).

Nginx are set to uploads of 128MB per file, it means he use up to 128MB of RAM for each uploading, once file are uploaded, its been send from Nginx RAM to Proxy server. If file are larger then 128MB, Nginx will cache it to disk, until your file is uploaded 100%, then Nginx will send the whole file to Proxy server, to FastCGI, who will get file, generate his hash and write it where it belongs. And i need this to say like that.

After i set up CGI configuration, its different story. Nginx still use 128MB, and if file are under 128MB, its fine, like i mention before. But if its larger, problem appears. Lets say you are uploading 1GB file, you are sending file to the server, Nginx will get you file to RAM, after 128MB of RAM for your upload are reached, your upload progress are paused(!), and you must wait, like 10-20sec, in that time, Nginx will send 128MB (supposed to cache it to the disk, not to send it right now) of your file to FastCGI, FastCGI will cache it to his own temp (drive) as 64MB (Nginx must wait at this point to send another half of a file), after Proxy caching it will write this 64MB of file where it belong (doesnt belong anywhere ATM, Proxy have to generate hash of a whole file first). Then Nginx sending another 64MB from RAM to Proxy, Proxy will cache it to the drive again, and then marge it with 64MB what he already got, in this story, compressing are enabled, files are getting two times copied/written, and then, your uploading progress are continued, you upload another 128MB to Nginxs RAM, and again, uploading paused, Nginx sending 128MB to Proxy, he gain 64MB to temp, then marge it, another 64MB, marge it, upload continue. Each time FastCGI get 64MB piece, he generate hash again and again, moving file from folder to folder. In the end when uploading are at 100%, you have to wait another 10sec for this to finish, and FastCGI will move file where it belongs (finally hash of a whole file), then, you will get download links... its just too slow for users (slow upload because its getting paused and slow getting download link (CGI write it to temp, hashing last 64MB piece, move to another folder, hash whole file, move to another folder where it belongs), and for server resources because of multiple I/O to drives and CPU usage for nothing.

There is no 64MB or 65000 numbers (except connections), not even by any calculation on Nginx or FastCGI, or Website, so IDK why FastCGI caching 64MB peaces in the first place, but i guess its about CGI configuration, because it uses his own temp, and this problem appear after i set FastCGI settings.

According to another sites on server, i see FastCGI are capable of using more RAM then its using for my upload site.

Here is my main Nginx and FastCGI configs:
Nginx: https://www.hastebin...rezuwikow.nginx
FastCGI: https://www.hastebin.com/aminowidon

Also need this (Nginx caching upload files; client_body_buffer_size and client_max_body_size get working) to prevent Slowloris attacks reaching Proxy servers.

Hope someone will know whats wrong. Thanks.

This post has been edited by Radius Nightly: 02 October 2017 - 06:02 AM


Is This A Good Question/Topic? 0
  • +

Replies To: File upload doesnt caching...

#2 ArtificialSoldier  Icon User is online

  • D.I.C Lover
  • member icon

Reputation: 1789
  • View blog
  • Posts: 5,700
  • Joined: 15-January 14

Re: File upload doesnt caching...

Posted 02 October 2017 - 10:28 AM

How do you know it's doing all that, and which settings did you change to make that happen? That link to your FastCGI config is the config for PHP, not FastCGI specifically.
Was This Post Helpful? 0
  • +
  • -

#3 Radius Nightly  Icon User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 50
  • Joined: 07-May 15

Re: File upload doesnt caching...

Posted 02 October 2017 - 02:13 PM

Because i was locally uploading files and downloading them all the time, tracking server what is he doing each time, like 10 hours a day im doing this for a month, its annoying and im tired of that, but easy thing to do. There was other little problems that i was tracking down too, but this one, IDK.
Yeah, settings for PHP, i have it configured (as you can see) when i was making everything up, long time ago, and everything worked fine (for my two sites, static one, and dynamic thats self-hosted), but, when i was setting services to run FastCGI on server, i forgot to set -c custom\path\php.ini, so it was working without any configuration all this time (probably with all defaults/no config), and because i didnt relay on any of that configs, when problems appear for my friend (for MySQL that im not using on my server), i find out that PHP configuration are not loaded, i fix it, so everything works fine (MySQL and other things), then i run some tests and find out Nginx doesnt caching big file uploads anymore (there was Evaluation version on site with 1GB file size limit that never gets deleted, and i notice its pausing every 128MB, and resuming after 10-20sec), thats when i started investigation to find out this problem. I tried changing lots of settings in Nginx, removing them, etc., and disabling temp/cache in PHP config, but nothing works, and playing with PHP config of 300 settings will take some time.

Its not only problem for Slowloris attacks, but could be also exploited.
A problem for max file size limit, on my upload site, when you choose file bigger then 128MB, he wont allow you to upload, you will get error about size limit. But if you turn off CSS, you can see Upload button, and hit it, so yeah, you will start uploading something bigger then 128MB. During upload, or after upload is done, PHP check filesize, if its larger then 128MB, he will delete it, and return error to you about file size limit, however, ATM (when this problem appear), you can pick up to 4GB file, get error about 128MB limit, disable CSS, hit Upload button, and it will get uploaded without any problem, giving you download links, like everything is fine. I just guess during upload, PHP cant check filesize because he is getting little pieces of 64MB, so max file size he get are 64MB that he is caching in temp, in the end when he make hash and set file where it belongs, its too late.

I can give you a link, so you can test it for yourself if its not against rules.
Was This Post Helpful? 0
  • +
  • -

#4 ArtificialSoldier  Icon User is online

  • D.I.C Lover
  • member icon

Reputation: 1789
  • View blog
  • Posts: 5,700
  • Joined: 15-January 14

Re: File upload doesnt caching...

Posted 02 October 2017 - 03:50 PM

I'm not sure how that would work, because PHP doesn't get executed until the file is finished uploading and the request is finished. When PHP starts is when it builds things like $_POST, $_FILES, $_SERVER, etc. If the max post size gets exceeded then $_POST and $_FILES will both be empty, so I'm not sure how that works at all to upload and process a 4GB file when the max post size for PHP is set to 128M. If PHP thought it was a 64MB file then I would imagine that's how big the file would be on disk. You can test that though, you can print the entire $_FILES array for example to see what's in there.
Was This Post Helpful? 0
  • +
  • -

#5 Radius Nightly  Icon User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 50
  • Joined: 07-May 15

Re: File upload doesnt caching...

Posted 03 October 2017 - 06:05 AM

You can upload like 9GB file, but FS/HDD are limited to 4GB, so he write down 4GB only, but actually there is no limit when CSS are disabled, i will fix it later, now i wanna fix caching. About tracking, just opening temp (Nginx and FastCGI, and watching like RAM (64MB) to HDD Cache 64MB, RAM clear, fill 64MB, write down via FastCGI...), Task Manager or Resource Manager to watch whats happening. Its clear whats happening (Nginx doesnt write anything to HDD, it just does not, but FastCGI are handsome writer, he write all notes down, he is a nice guy).

I pull out files from backup, change some little things, and running tests on my PC. Now i may mix some informations. As far as i figure out for now, again the same, but i think its about Nginx. Nginx does not caching big uploads (with or without php.ini, he is sending to FastCGI directly), only caching downloads (from FastCGI).

Without php.ini, files under 128MB are cached in RAM by Nginx. Files larger then 128MB are cached in Nginx RAM as 64MB and then sent to FastCGI. Slow I/O transfer ~5MB/sec that i like. IDK why 64MB, i tried setting write, temp, output and all other things like noom from "k" like "512k" to "m", same as 4x256k into 32x256m and other stupid things, but its still 64MB Nginx Cache in RAM.
With php.ini, files under 128MB are not cached in any way (actually around 800K in RAM) by Nginx (even its configured to do so), and sent to FastCGI directly. Gives fast I/O ~100MB/sec with pausing transfer (on this PC its 2sec pause, but on server its 20sec pause between).

Will try some more things and i will come with more information soon.
Was This Post Helpful? 0
  • +
  • -

#6 Radius Nightly  Icon User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 50
  • Joined: 07-May 15

Re: File upload doesnt caching...

Posted 03 October 2017 - 12:50 PM

Guess i figure out whats happening. Looks like its FastCGI, and not Nginx (but still have to make client body buffering cached to the drive: will check it out).

I added some options to Nginx (most of them are default so shouldnt be there, but whatever), in the end i increase file size (Nginx - client_body_buffer_size, PHP - post_max_size, upload_max_filesize and memory_limit) to 512MB.

So, files under 512MB are cached in RAM by Nginx, after upload are finished, its sent to FastCGI, works fine, as usually.
If uploading file are bigger then 512MB, its cached (half of 512MB limit, but thats because of PHP) in RAM by Nginx (thats 256MB caching in RAM), and each uploaded 256MB of file are sent from Nginx RAM to FastCGI who caching it to the drive/temp as 256MB piece (this process pausing uploading process every 256MB).

And i search for this (limit of 128MB caching 64MB pieces: limit of 512MB caching 256MB pieces, it was obviously half), and according to this http://data.agaric.c...-php-and-apache it looks like PHP smart feature or whatever.

Quote

(example for 16MB files)

upload_max_filesize = 16M
post_max_size = 18M

you get:
- no more that what upload_max_filesize says
- no more than half of post_max_size

So, the right numbers for up to 32MB files seem to be:

upload_max_filesize = 32M
post_max_size = 64M


Also error for file size limit after disabling CSS start working (somehow).


Will take a look into Nginx body caching now. He never cache anything to client_body_temp, i guess he does only when RAM are full, but i want him to cache it when file size are above the size limit, instead of sending it as little pieces to FastCGI (half of a size limit) pausing upload.
Was This Post Helpful? 0
  • +
  • -

#7 ArtificialSoldier  Icon User is online

  • D.I.C Lover
  • member icon

Reputation: 1789
  • View blog
  • Posts: 5,700
  • Joined: 15-January 14

Re: File upload doesnt caching...

Posted 03 October 2017 - 12:57 PM

I don't see that behavior on our servers. We have the max upload filesize set to 400M and the post max size set to 410M, and people can successfully upload files up to 400M. It wouldn't make sense if post max size was cut in half, that sounds like a bug. It might be a problem with FastCGI, we're not using that as our handler. If it's taking multiple seconds to flush RAM to disk it sounds like you've got some I/O problems, that sounds like the slowest disk to ever disk. I wonder if these issues aren't due to bugs in either FastCGI or Nginx.
Was This Post Helpful? 0
  • +
  • -

#8 Radius Nightly  Icon User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 50
  • Joined: 07-May 15

Re: File upload doesnt caching...

Posted 03 October 2017 - 02:01 PM

Yeah, its still successful, but IDK why in half limit to write (only if file exceed file size limit). Maybe its good thing (like not to overload server by one big upload), but Nginx doesnt allow upload progress to continue until he send half of a file size limit (if limit are exceeded) from RAM to FastCGI, and until he write it down to the drive, then upload may continue, its 2sec, but on server its 10-20sec pause because thats not the only thing to do.

For your settings its like 400MB limit, if im gonna upload 500MB file, will first upload 200MB piece, pause, then 200MB piece, pause, then 100MB piece instead of caching it directly (or even using 400MB of RAM for transfer).

I just set Nginx client_body_buffer_size to 4m, and upload file of 800MB, so he cache it to client_body_temp (using 4MB of RAM for transfer), but once he reach 256MB with cached file piece, its been sent to FastCGI (limit 512MB). Thats two different processes, is it possible that FastCGI are pulling by itself? I dont see any reason why would Nginx sending half piece of a PHP limit when he should obey his configs.
Was This Post Helpful? 0
  • +
  • -

#9 ArtificialSoldier  Icon User is online

  • D.I.C Lover
  • member icon

Reputation: 1789
  • View blog
  • Posts: 5,700
  • Joined: 15-January 14

Re: File upload doesnt caching...

Posted 03 October 2017 - 02:06 PM

Quote

For your settings its like 400MB limit, if im gonna upload 500MB file, will first upload 200MB piece, pause, then 200MB piece, pause, then 100MB piece instead of caching it directly (or even using 400MB of RAM for transfer).

Right, but we don't see that on our server. There's no pausing. If the file is too large then the server knows that, there's nothing going on with pauses or pieces.

Quote

I dont see any reason why would Nginx sending half piece of a PHP limit when he should obey his configs.

I don't either, that's why it sounds like a bug.
Was This Post Helpful? 0
  • +
  • -

#10 Radius Nightly  Icon User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 50
  • Joined: 07-May 15

Re: File upload doesnt caching...

Posted 03 October 2017 - 02:41 PM

After changing PHP memory limit, max post and file size (doesnt have effect, i leave them all at 128MB, and leaving Nginx at 4MB), setting limit to 1024MB at php/site, Nginx was able to cache 1GB file on disk, sending it to FastCGI who capture whole file at once. Uploading file bigger then 1GB, Nginx caching into 512MB pieces and sending it to FastCGI.

Guess its how FastCGI handle uploading files who exceed file size limit set in php of a site (cutting them into half of a limit and work with pieces). Thats how it looks like to me and will live with that, even there is no any kind of true documentation for this. Since im using FastCGI from 2012, probably a bug.
Or you know how developers say, its not a bug, its feature. : D

Thanks.

This post has been edited by Radius Nightly: 03 October 2017 - 02:42 PM

Was This Post Helpful? 0
  • +
  • -

#11 ArtificialSoldier  Icon User is online

  • D.I.C Lover
  • member icon

Reputation: 1789
  • View blog
  • Posts: 5,700
  • Joined: 15-January 14

Re: File upload doesnt caching...

Posted 03 October 2017 - 04:24 PM

Quote

Guess its how FastCGI handle uploading files who exceed file size limit set in php of a site (cutting them into half of a limit and work with pieces).

Are you saying it does that during the actual upload? Because it doesn't know how big the file is going to be until it's finished, how does it know whether or not it's over the limit? Posts requests do have a content-length header, but that's for the entire request instead of the file size. You can upload multiple files, for example, so the content-length header wouldn't be an indication of how big any of the files are, just the total request body.

Moreover - I don't think that FastCGI has any knowledge of PHP settings at all. Those settings are used by PHP, not FastCGI.
Was This Post Helpful? 0
  • +
  • -

#12 Radius Nightly  Icon User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 50
  • Joined: 07-May 15

Re: File upload doesnt caching...

Posted 04 October 2017 - 05:00 AM

Yeah. Its single file upload, with POST method file size, name, date, etc. are sent. Was testing Nginx to convert anything to GET while testing, didnt have effect. When file exceed limit, zero are returned. So if you upload 4GB file, FastCGI or website/php (and Nginx with different setting) will mark it with content-length 0 (maybe because of that he deal upload with half limit), but when you downloading, Nginx or/and FastCGI will gather true content-length for your web-browser. Thats not something i care about.

FastCGI settings are by default called php.ini, thats what i mean by FastCGI (php.ini for FastCGI), and when i say website/code/php, then i mean php code from the site, like imfunction.php. Sorry if i mix this two somewhere.
Was This Post Helpful? 0
  • +
  • -

#13 no2pencil  Icon User is online

  • Professor Snuggly Pants
  • member icon

Reputation: 6543
  • View blog
  • Posts: 30,648
  • Joined: 10-May 07

Re: File upload doesnt caching...

Posted 04 October 2017 - 05:59 AM

I would put wireshark on something higher in the network & validate what is causing the pause. Either the client or networking equipment is throttling the connection. I would think it's not likely the server, as with those settings (& as ArtificialSoldier has said), php/nginx is just going to take what it's given.
Was This Post Helpful? 0
  • +
  • -

#14 ArtificialSoldier  Icon User is online

  • D.I.C Lover
  • member icon

Reputation: 1789
  • View blog
  • Posts: 5,700
  • Joined: 15-January 14

Re: File upload doesnt caching...

Posted 04 October 2017 - 10:16 AM

There might be confusion with the terminology. FastCGI and PHP are completely separate pieces of software. php.ini is not the configuration file for FastCGI, it is the configuration file for PHP. FastCGI has its own configuration file (e.g. /etc/nginx/fastcgi.conf). FastCGI is the piece of software between the web server and PHP, it's the way that the web server executes PHP (the "gateway", that's the "G" in "CGI"). You have a web server (Nginx), a gateway (FastCGI), and the module (PHP). The web server handles the entire request and response, and if the request matches the web server configuration settings then the web server will pass the request to the gateway, which passes it to PHP. PHP (and, as far as I know, FastCGI) do not even begin handling the request until the web server has received the entire thing. Once the web server gets the entire request then it passes it off to the other software to process the request and return the response. If you're measuring things that are happening during the actual upload, then that's the web server. That's not FastCGI and it's not PHP. The other things only get involved once the web server has received the entire request. After PHP has started up, but before it starts executing your PHP code, it processes the request to build things like $_POST and $_FILES. All of this RAM caching and disk I/O that you seem to be measuring sounds like you are measuring those things while the actual request is still being sent to the server, which at that point has nothing to do with PHP, and probably nothing to do with FastCGI yet either. That's part of why I was confused, I thought you had some debugging hooks somewhere in Nginx and the actual FastCGI gateway and you were looking at all of these I/O operations going back and forth, but it sounds like when you say FastCGI you are referring to PHP. And, just to be clear, when I say PHP I'm referring to the actual PHP software, not any PHP code that you wrote. PHP is a program that takes your code and executes it, and I'm referring to the actual PHP program.
Was This Post Helpful? 0
  • +
  • -

#15 Radius Nightly  Icon User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 50
  • Joined: 07-May 15

Re: File upload doesnt caching...

Posted 04 October 2017 - 11:07 AM

I understand what you saying, but i was thinking about FastCGI with and without php.ini, because i dont know where is the problem (that appear after using php.ini). Confusing myself right now, but i got it.

Was measuring while upload are in progress, and it make sense because no one can pull anything from Nginx (specially if its against any possible configuration).
However this is situation where all possible file limits are exceeded (i dont think anyone testing this lol). I dont think its about throttling, because its pausing for 2sec on SSD locally (to write it down and resume; shouldnt do that), only with big files it can be noticed (small files are just too fast to notice, even with small numbers he is doing same thing, sending small pieces of big file to FastCGI constantly, cant notice it (no pauses), but under eg. Slowloris attack thats bad thing to do, because all FastCGIs can get locked down).

Maybe its another Nginx bug, im using version 1.13.4 (tried 1.13.5) for Windows that already got some bugs and limitations (can work around most of them), will ask Nginx if they got reported any bug about this, cant find any bug list or something like that, except this for Windows version:
-Although several workers can be started, only one of them actually does any work.
-A worker can handle no more than 1024 simultaneous connections. (OS limit, can be fixed by TCPIP)
-The UDP proxy functionality is not supported.
Was This Post Helpful? 0
  • +
  • -

  • (3 Pages)
  • +
  • 1
  • 2
  • 3