9 Replies - 247 Views - Last Post: 22 October 2019 - 03:21 PM

#1 bobsmith76   User is offline

  • D.I.C Regular

Reputation: 11
  • View blog
  • Posts: 383
  • Joined: 14-February 17

Can a remote VM do calculations on data that is on another computer?

Posted 15 October 2019 - 01:39 AM

My computer is too slow. It might take me up to two weeks to do the calculations I want to do so I have decided to try my hand at remote programming, that is to say, dividing my work amongst several computers. To do this it seems that I need to SSH into a virtual machine. Here's my question: do I have to transfer the data that it is going to process onto that machine? I have a terrabyte of data and I'm really worried that transferring the data will take up too much time, even though if I have ten computers I only need to transfer 100 gigs onto each computer. Still, if I transfer it 100 gigs to one remote computer, I will then have to delete the data when it's done, which also takes time, though not that much. With my laptop I can do calculations on an external hard drive even though the python script is on the laptop and I'm hoping that the remote virtual machine can do the calculations by accessing the data on my laptop without have the data on its harddrive.

Is This A Good Question/Topic? 0
  • +

Replies To: Can a remote VM do calculations on data that is on another computer?

#2 no2pencil   User is online

  • Professor Snuggly Pants
  • member icon

Reputation: 6822
  • View blog
  • Posts: 31,459
  • Joined: 10-May 07

Re: Can a remote VM do calculations on data that is on another computer?

Posted 15 October 2019 - 04:55 AM

My initial reaction is sure, but without knowing how you are doing 'calculations' it's difficult to make any technical suggestions.

Of course, anything that you are doing over ssh, you could do with a shell script via cron, or send to the background with &, or even run in screen. That way it can crunch numbers while the session is closed.
Was This Post Helpful? 0
  • +
  • -

#3 bobsmith76   User is offline

  • D.I.C Regular

Reputation: 11
  • View blog
  • Posts: 383
  • Joined: 14-February 17

Re: Can a remote VM do calculations on data that is on another computer?

Posted 15 October 2019 - 06:38 AM

Well let me be more specific. On computer 1 we have the file:

computer1.py
hey = 0



On computer two we have:

you = open_file('computer1.py')
you += 1
print (you)



Something like that. Even better would be instead of line 3 in the second code:

save_file(you, 'computer1.py)



Is that possible?

This post has been edited by bobsmith76: 15 October 2019 - 06:38 AM

Was This Post Helpful? 0
  • +
  • -

#4 modi123_1   User is online

  • Suitor #2
  • member icon



Reputation: 15318
  • View blog
  • Posts: 61,423
  • Joined: 12-June 08

Re: Can a remote VM do calculations on data that is on another computer?

Posted 15 October 2019 - 06:54 AM

Fun question - are these remote machines, VMs, what evers.. on a local network or not?
Was This Post Helpful? 0
  • +
  • -

#5 no2pencil   User is online

  • Professor Snuggly Pants
  • member icon

Reputation: 6822
  • View blog
  • Posts: 31,459
  • Joined: 10-May 07

Re: Can a remote VM do calculations on data that is on another computer?

Posted 15 October 2019 - 07:13 AM

if you want to access the server with a function that accesses at the file level, my immediate thought would be to remote mount the file system. The question then becomes which is slower, the processing or the bandwidth?

Why not put that processing file onto the server where the files are located? Otherwise, you may have to replace open_file() with something like open_socket()
Was This Post Helpful? 0
  • +
  • -

#6 bobsmith76   User is offline

  • D.I.C Regular

Reputation: 11
  • View blog
  • Posts: 383
  • Joined: 14-February 17

Re: Can a remote VM do calculations on data that is on another computer?

Posted 15 October 2019 - 01:00 PM

View Postno2pencil, on 15 October 2019 - 02:13 PM, said:

if you want to access the server with a function that accesses at the file level, my immediate thought would be to remote mount the file system. The question then becomes which is slower, the processing or the bandwidth?

Why not put that processing file onto the server where the files are located? Otherwise, you may have to replace open_file() with something like open_socket()


Because the files occupy one terabyte. At the current rate it will take me 2 to 4 weeks to process them. For this reason I'm trying to get about 20 remote VMs to each process 1/20th of the files.

View Postmodi123_1, on 15 October 2019 - 01:54 PM, said:

Fun question - are these remote machines, VMs, what evers.. on a local network or not?


Modi, I still don't know if you're the webmaster of this site or not. If you are, let me give you some unsolicited advice: are you aware that it is snide remarks like these that drive users away from your site? If you're trying to get more people to use your website then know that it is snide, unhelpful remarks like these that inhibit you obtaining that goal.
Was This Post Helpful? 0
  • +
  • -

#7 modi123_1   User is online

  • Suitor #2
  • member icon



Reputation: 15318
  • View blog
  • Posts: 61,423
  • Joined: 12-June 08

Re: Can a remote VM do calculations on data that is on another computer?

Posted 15 October 2019 - 01:04 PM

Your snide detector is off. That was not to be snide, but feel out how you plan on doing things as there is a large detractor.

It was a side question to your current thread with the end goal of pointing out most ISPs have a terabyte cap for monthly data usage. If you plan on sending out terabytes of data you will butt onto that cap pretty quickly. This could mean data speed limiting, overage charges, etc. Even with "unlimited" what evers - terabyte is a standard ceiling before crossing into the danger zone.
Was This Post Helpful? 0
  • +
  • -

#8 no2pencil   User is online

  • Professor Snuggly Pants
  • member icon

Reputation: 6822
  • View blog
  • Posts: 31,459
  • Joined: 10-May 07

Re: Can a remote VM do calculations on data that is on another computer?

Posted 15 October 2019 - 03:53 PM

View Postbobsmith76, on 15 October 2019 - 04:00 PM, said:

View Postno2pencil, on 15 October 2019 - 02:13 PM, said:

if you want to access the server with a function that accesses at the file level, my immediate thought would be to remote mount the file system. The question then becomes which is slower, the processing or the bandwidth?

Why not put that processing file onto the server where the files are located? Otherwise, you may have to replace open_file() with something like open_socket()


Because the files occupy one terabyte. At the current rate it will take me 2 to 4 weeks to process them. For this reason I'm trying to get about 20 remote VMs to each process 1/20th of the files.

I got that the first time that you pointed it out. Flip the process. Copy the scripts where your data is.
Was This Post Helpful? 0
  • +
  • -

#9 ndc85430   User is online

  • I think you'll find it's "Dr"
  • member icon

Reputation: 981
  • View blog
  • Posts: 3,868
  • Joined: 13-June 14

Re: Can a remote VM do calculations on data that is on another computer?

Posted 16 October 2019 - 12:39 AM

You haven't mentioned what these calculations are for, or where this large quantity of data comes from. What it makes me think of is the large scale simulations that some of my colleagues would do when I was doing my PhD in physics (though in that case, the large amounts of data were output, rather than input, but whatever). For such tasks, we had access to high performance computing (HPC) facilities both at our university and elsewhere. I don't suppose you're affiliated with any university and could find out what HPC facilities are available to you?
Was This Post Helpful? 0
  • +
  • -

#10 bobsmith76   User is offline

  • D.I.C Regular

Reputation: 11
  • View blog
  • Posts: 383
  • Joined: 14-February 17

Re: Can a remote VM do calculations on data that is on another computer?

Posted 22 October 2019 - 03:21 PM

View Postndc85430, on 16 October 2019 - 07:39 AM, said:

don't suppose you're affiliated with any university and could find out what HPC facilities are available to you?

Thanks for your interest. No I don't. But I'm pretty sure Google Data Store will allow me to do what I want. I have to put my data in a nosql database. I'll post again if that does not work.

thanks I appreciate your answer
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1