8 Replies - 1002 Views - Last Post: 22 October 2013 - 09:19 PM

#1 cbt13  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 2
  • Joined: 28-October 12

Opinions on what language to use for my project.

Posted 19 October 2013 - 05:33 PM

Hi all,

I wrote a program in AutoHotkey that downloads millions JSON files, parses them, and creates new JSON files according to the data. The way I do this, I create 1000s of AHK scripts that downloads only a portion of JSON files and run them, say, 100 at a time. This proved to be unpractical.

I'm looking for a new language that I can convert my project to. It should be able to download multiple files simultaneously (threading?) and should be able to parse/edit multiple files simultaneously.

I once tried to use C/C++ for it but I had problems interacting with URLs (I'm using SteamAPI to get the JSONs). And I don't even know if I can do the job as fast as AHK is doing it right now.

Any opinions?

Is This A Good Question/Topic? 0
  • +

Replies To: Opinions on what language to use for my project.

#2 AVReidy  Icon User is offline

  • D.I.C Regular
  • member icon

Reputation: 65
  • View blog
  • Posts: 431
  • Joined: 17-February 11

Re: Opinions on what language to use for my project.

Posted 19 October 2013 - 06:09 PM

Go for concurrency & speed

This post has been edited by AVReidy: 19 October 2013 - 06:14 PM

Was This Post Helpful? 0
  • +
  • -

#3 baavgai  Icon User is offline

  • Dreaming Coder
  • member icon

Reputation: 5881
  • View blog
  • Posts: 12,758
  • Joined: 16-October 07

Re: Opinions on what language to use for my project.

Posted 19 October 2013 - 06:29 PM

Wait, Go is still around? Fascinating.

The first language that comes to mind... wait for it... is Scala. I'm not kidding. I mean, if it can run Twitter.

Reasoning is, you want a language with all the packages in place. Grabbing urls, text process, etc. For that, Python would be my first choice. With C# or Java runners up. However, lots of threads, you want a language that natively does that kind of work for you. The more functional kids, with zero side effect mentalities, tend to excel at this. However, you don't want to loose the canned libraries those other languages offer. So, rather than C# or Java, F# or Scala.

Scala has an actor metaphor built in and a JVM is more broadly available, so it seems a reasonable choice.
Was This Post Helpful? 0
  • +
  • -

#4 BlueMelon  Icon User is offline

  • D.I.C Head

Reputation: 40
  • View blog
  • Posts: 187
  • Joined: 27-April 10

Re: Opinions on what language to use for my project.

Posted 19 October 2013 - 07:19 PM

I would definitely pick python for this type of script. Quick, easy and fast.
Was This Post Helpful? 0
  • +
  • -

#5 Adak  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 331
  • View blog
  • Posts: 1,168
  • Joined: 01-April 11

Re: Opinions on what language to use for my project.

Posted 19 October 2013 - 07:19 PM

Go is coming along nicely. I would advise Scala however - a more mature (and quite impressive) language. There's a very good vid on YouTube about it.

If you need help with Scala though, is that readily available on the net?

From using AHK, do you have a feel for where the largest bottleneck is? Input? Processing the files? Output?

If processing is the problem, then perhaps the file parsing, etc. algorithm could be improved. Perhaps AHK gets the files in one process, and C does the processing in another?

Lots of help with C, around. :helpsmilie:/>

This post has been edited by Adak: 19 October 2013 - 08:22 PM

Was This Post Helpful? 0
  • +
  • -

#6 Adak  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 331
  • View blog
  • Posts: 1,168
  • Joined: 01-April 11

Re: Opinions on what language to use for my project.

Posted 20 October 2013 - 06:27 PM

Can you give us more info on the problem? How long does it take currently to run the program? How many lines does it process? What is your OS? (Windows 7?)

Do you have a (hard) deadline for getting this done?

With more understanding of the problem, we could help get the parts of it tuned up. Then, with one instance of it running better, every parallel process could run better, as well.

I can't say how it is with your program, but string work (parsing), is not usually done optimally.
Was This Post Helpful? 0
  • +
  • -

#7 xclite  Icon User is offline

  • LIKE A BOSS
  • member icon


Reputation: 911
  • View blog
  • Posts: 3,178
  • Joined: 12-May 09

Re: Opinions on what language to use for my project.

Posted 20 October 2013 - 07:17 PM

I'm just here to second that I would think Go or Scala are both excellent choices.
Was This Post Helpful? 0
  • +
  • -

#8 Adak  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 331
  • View blog
  • Posts: 1,168
  • Joined: 01-April 11

Re: Opinions on what language to use for my project.

Posted 22 October 2013 - 05:45 PM

@cbt13:

Is there any news on your project? Is there any way to follow the project?

It's frustrating when a good project comes along, and then < Poof! > you can't follow it any more.
Was This Post Helpful? 0
  • +
  • -

#9 Skydiver  Icon User is online

  • Code herder
  • member icon

Reputation: 3616
  • View blog
  • Posts: 11,269
  • Joined: 05-May 12

Re: Opinions on what language to use for my project.

Posted 22 October 2013 - 09:19 PM

I finally had a few minutes to peek at AutoHotKey. From my brief scan, it looks like it doesn't have any native JSON parsing built in and that the SteamAPI calls that are made need to go through several layers. So simply moving away from the interpreted environment of AutoHotkey/AutoIt to a compiled or a compiled just in time version of code will already give significant speed boost just for the parsing of the JSON. If you can also make the web service calls directly instead of having to go through the SteamAPI, then even more speed can be gained.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1