Yup, cURL did it. They must not like httpd requests or whatever. Can't say I blame them. Especially from a random Amazonian IP.

Execution time down from 2115 seconds to 51 seconds.

Thanks for the tip. The entire cloud-based backend is functional now!

It's Amazon EC2, so I have full control over what I install on (in this case) Ubuntu Server. I happen to know that cURL is installed. So, I'll try that!

Their API is designed well enough, although it's not really a full API...more like "format your URL like this and you get a JSON yay!" The problem is that their database is not really laid out sequentially. You'd think it would start at item #0 and work its way up to somewhere in the ~37.000 range (from what I hear, that's how many they've got), but no, the numbers assigned to different entries are based on random stuff like how long it's been around (smaller numbers) and more recently its "category," which just further complicates things as it's all very closed-source.

They probably are throttling my requests, which is weird because I'm making 66 or so whereas sites (read: fansites) that maintain what are essentially mirror databases of their own will make over 37.000 a day.

But yes, it's worth mentioning that while I was in the process of writing the retrieval script, to make sure it worked I made a test JSON with one item and it would always take between 30 and 32 seconds to get data just for one.

Interestingly, I'd done the exact same thing in Java and it would take seconds to retrieve data for the whole list!

Thing is, I want to automate the creation of a master JSON because the Java program was querying the database each time I selected an item from the list, resulting in hundreds of requests instead of a flat number.

Also the value I want to retrieve only changes once every 24 hours anyway.

Hmm.

I'll have to ask the controlling company about their throttling policy. Maybe I should use cURL? In any case, I'm a moderator of their official forums, which is an invite-only position, so hopefully they'll know that I'm not out to write any kind of malicious code!

Thanks for the tips. I'll try the sleep(1) anyway, just to see if it has any effect.

Okay, thank you!

I will simply have to learn more about JSON, then. It's not that bad as is - I simply compiled the org.json package from, well, json.org...just gotta figure out a working method of obtaining the specific string (just an integer) that we need and then we're off.

I certainly have no qualms about foregoing XML altogether. I know a fair bit about it through using BOINC, as any and all end-user customisation or tweaking is done via the editing or creation of XML files, which are then read in Perl and the rest is a mystery to me.

The site from which we are pulling data is written almost entirely in Javascript. In fact, I think it IS entirely Javascript, which would explain the long load times, and also why we want to keep our own centralized database to reduce querying them all the time.

JSON it is, then! Thank you for the tips, as always.

I promise this program has nothing at all to do with prime numbers.

Got it. Thanks! I will do my homework and see what I can cook up with Qt!