9 Replies - 719 Views - Last Post: 30 August 2017 - 12:52 AM

#1 wouterrem  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 25-July 17

Server architecture for online calculation platform

Posted 25 July 2017 - 06:34 AM

Hi everyone,

I'm looking for advice on how to build our server architecture!

At our startup, we're building an online aerodynamics platform.
We're at the beginning of development (apart from the aero scripts), which means we have to determine an architecture & choose programming languages for each block.
After talking to a number of developers, the idea would be the following:

Back-end
- General language
Use python (instead of php for example) as it is widely used and scalable (we'll be doing a lot of simulations, hopefully ).
Is that a good choice (risky to ask in a python forum, I know)?
- Data
Each calculation generates a lot of data (some GB's). Are there specific structures/tools/compression methods to manage this, or just drop it on the server?
Should we use MySQL for example to manage user data (also the small data, like project names, user details, ...)?
Is that compatible with python?
- Encryption / authentication
To provide comfort regarding confidentiality, we should encrypt data and provide secure login.
Are there specific tools for that (preferrably in a python environment)?
- API's
Easy to write in python?
- Framework
Is Django a good framework to use? Or is this something for people with limited experience, and should you code everything yourselve to build it properly?
- What other tools should we consider (nginx, apache, ...)? Things are new to me!

Aero
- Our scripts modify text files using m4 and mustache. Is that a good choice or are there better alternatives?

Front-end
- Framework
Would Angular / react be a good choice to put on top of Python/Django?
Is there something in between to let them communicate?
- 3D
We would use WebGL (Three.JS): does anyone have experience with this?
Blend4web could be an alternative: anyone know this one?



General:
- Are there good visuals available on how all these blocks work together (back-end layer, front-end layer, ...)?
Or perhaps even a configurator?
- Will it pay off to have python as the preferred language throughout the company?
Or am I focusing too much on this (and will I select non-ideal solutions just because they're python based)?

Thanks a lot for all your tips to get us started!!

Wouter.

Is This A Good Question/Topic? 0
  • +

Replies To: Server architecture for online calculation platform

#2 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 13485
  • View blog
  • Posts: 53,847
  • Joined: 12-June 08

Re: Server architecture for online calculation platform

Posted 25 July 2017 - 07:06 AM

Hmm. Are you the only one doing this 'architecture' work?
Was This Post Helpful? 0
  • +
  • -

#3 wouterrem  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 25-July 17

Re: Server architecture for online calculation platform

Posted 25 July 2017 - 07:46 AM

View Postmodi123_1, on 25 July 2017 - 07:06 AM, said:

Hmm. Are you the only one doing this 'architecture' work?


Hi Modi,

no, on the contrary, we're still looking for someone to join the team for this!
And I thought it would be good to have some thoughts on the architecture and programming languages first, so I can look for people in a more efficient way.

So what would be your answer to all of the questions?

thanks!

Wouter.
Was This Post Helpful? 0
  • +
  • -

#4 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 13485
  • View blog
  • Posts: 53,847
  • Joined: 12-June 08

Re: Server architecture for online calculation platform

Posted 25 July 2017 - 07:54 AM

Well, I am not going to spec out your entire project for you, but I would start with the basics: creating data and retaining data. Those should be the nut of the issue.
How is it generated?
What format?
How much?
How long?
Is it even being generated on this machine?
Is it even being stored on this machine?
Should there be some web access to it?
.. and so on.

Focus the design on the important part and not based around some language or framework.
Was This Post Helpful? 0
  • +
  • -

#5 NeoTifa  Icon User is offline

  • NeoTifa Codebreaker, the Scourge of Devtester
  • member icon





Reputation: 4081
  • View blog
  • Posts: 18,152
  • Joined: 24-September 08

Re: Server architecture for online calculation platform

Posted 25 July 2017 - 11:30 AM

If it's massive amounts of data, Python is a standard programming language, but I know nothing of writing server-side stuff with Python (probably why I got a D in that class).
Was This Post Helpful? 0
  • +
  • -

#6 jon.kiparsky  Icon User is online

  • Chinga la migra
  • member icon


Reputation: 10682
  • View blog
  • Posts: 18,295
  • Joined: 19-March 11

Re: Server architecture for online calculation platform

Posted 25 July 2017 - 12:13 PM

A few thoughts for you:

  • Python is a perfectly reasonable choice for back-end work. You don't want to do heavy math in python, of course, but there are very good scientific computing libraries available, which are extremely well supported and quite fast. It's also a good choice for handling HTTP requests, including delivering pages and API work. For the latter, the obvious things to consider are Django and flask. Both are very popular web frameworks, the main difference is that django is a heavyweight framework which is strongly geared to its original use case, which is a newspaper site in Kansas. It turns out that there's a LOT of stuff in that bailiwick, and a lot of things can be close enough to it, but as you get away from managing articles and reporters and readers and comments, you find yourself fighting the platform a little. Flask, on the other hand, is a featherweight framework. It basically offers nothing, and doesn't get in your way at all - but you have to do more to get it up and running, and it's maybe less obvious how to work with it.
  • In this case, either would probably work fine. Ruby on Rails or sinatra - which more or less correspond to Django and flask, but in ruby - would also work fine.
  • All of the above can handle your MySQL, along with other flavors of SQL. Some will do NoSQL as well, but not all.
  • As one obvious suggestion, I would consider breaking out the calculations into a separate piece. So you have your server architecture that accepts requests and relies on some other service to do the computations. Since the handling of requests has nothing at all to with aerodynamic calculations, this is a natural seam, and making the cut at this seam from the start will be a lot easier than cutting later. Among other things, this will allow you to do stuff like spinning up separate instances of the calculation engine as needed (since each calculation will presumably be independent of other calculations, this is an obvious step)
  • It might even be a good idea to develop the two pieces in different languages - python for the calculations, RoR or sinatra for the request handler - in order to make sure the coupling between them is only at the intended connection points.

Was This Post Helpful? 0
  • +
  • -

#7 Skydiver  Icon User is online

  • Code herder
  • member icon

Reputation: 5890
  • View blog
  • Posts: 20,102
  • Joined: 05-May 12

Re: Server architecture for online calculation platform

Posted 26 July 2017 - 04:34 AM

As complete outsider but with a tiny bit of curiosity into computational fluid dynamics, my understanding is that computing things for aerodynamics tends to be very CPU and memory intensive. If your plan is to provide such services to a lot of people, the suggestion above to decouple at the point of getting the web requests and returning the computed results from the actual computation is right on the money. I would also start thinking about how to setup the backend so that it can elastically scale out as needed. Obviously, you'll need to balance that out with the cost of having multiple servers.
Was This Post Helpful? 0
  • +
  • -

#8 wouterrem  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 25-July 17

Re: Server architecture for online calculation platform

Posted 26 July 2017 - 04:54 AM

Hi all,

thanks for the input!

Short bit of feedback:
indeed, the calculation itself is done in a separate (open source) framework.
Basically, we can start a calculation with a single command line, with results as the output.
So a clean cut is definately possible.

And yes, for each calculation, a separate virtual machine should be started.
Web server, data server, etc. should be on continously, but for an incoming calculation request, a new machine should be spun up.
On that machine, the calculation is started using that single command line.
Once it's finished, data should be sent back to the data server.

The use of python not only concerns the interfacing (which indeed can be cut), it also concernts people within the small team being able to talk in the same language.
Does it make sense that we try to do everything as much as possible in the same language?
Aero scripting, back-end work, ... so that if someone would fall ill, or leaves the company, we can take over?
Or is that a second order reason to choose a language and should I first focus on it's suitability to build the back-end?

If we look at it from a pure technical point of view:
What is the best language to build the back-end for this application?
And which building blocks / frameworks would you recommend?
For example, a ranking in your opinion would be nice! (e.g. 1. Ruby, 2. python, 3. php)


thanks!

Wouter.
Was This Post Helpful? 0
  • +
  • -

#9 ssraj  Icon User is offline

  • New D.I.C Head

Reputation: 2
  • View blog
  • Posts: 23
  • Joined: 02-December 15

Re: Server architecture for online calculation platform

Posted 29 August 2017 - 11:40 PM

Here are the suggestions,

- For the front-end development purposes, both Angular.JS and react.js are responsive. In Angular.Js, you can use CSS, HTML, and Javascript for the development. Where in React.JS, you will get to use a Syntax called JSX which allows using the HTML code.

- For the backend, you can use either Node.JS or Python. Python is an old compared to Node.JS. But, Node.js is the recommened by community as the Javascript is the future of technologies and easy to learn. Node.js is extremely fast and the data communication with a server will enhance user experience.

- If you are willing to build real time application like Chat application or instant calculation than you can choose node.js. Other than that if you need to use streaming classes than also Node.JS will run at the stunning pace.

- Node.JS also gives the fastest database operations with NO-SQL database like Mongo.db where you only need to create object of the database and run a file. To perform various database operations, just call the functions which are already exist in the library.
Was This Post Helpful? 0
  • +
  • -

#10 wouterrem  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 25-July 17

Re: Server architecture for online calculation platform

Posted 30 August 2017 - 12:52 AM

Thanks for the input SSRAJ!

So basically, you would recommend NodeJS over python?
I've been told it requires more discipline to write proper and stable code, as the syntax allows for more errors.
Would you agree on that?

If we go for Node, which framework would you recommend? ExpressJS seems to be rather popular.
(or is that not a framework and am I mixing up things?).

And finally, where do you think I could source a (very) good full stack Node/Javascript developer?
One side note is that non-EU members are quite a bit more difficult to hire...but it might be worth considering, you never now.

Thanks a lot!

PS:
you can also reach me at [email protected]
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1