10 Replies - 13302 Views - Last Post: 15 July 2009 - 08:27 AM Rate Topic: -----

#1 renegade334  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 9
  • Joined: 13-July 09

Program to collect data from a website

Posted 13 July 2009 - 09:31 PM



Hey,

I am an economics student and am trying to code a program which will collect data from a website. I essentially need to collect data from auction websites, so I can use the results, perform some regression and mathematical analysis (stata) for my masters thesis.

This could be done in any of these ways
1. Collected data stored in .txt/.xls files
2. A program that will merely take screenshots of an entered page every x minutes, so I can do the data entry myself.

I essentially need pointing in the right direction, so I can do (attempt) some coding. Some help would greatly be appreciated!

Alternatively, if I don't sound too lazy, is there any software available which already does this? ( though I'd like to do the coding)

Thank you.


This post has been edited by renegade334: 13 July 2009 - 09:32 PM


Is This A Good Question/Topic? 0
  • +

Replies To: Program to collect data from a website

#2 Dogstopper  Icon User is offline

  • The Ninjaducky
  • member icon



Reputation: 2870
  • View blog
  • Posts: 11,021
  • Joined: 15-July 08

Re: Program to collect data from a website

Posted 13 July 2009 - 09:53 PM

If it were me, I would go to some of the auction websites to see if they have some public APIs or places that you could connect to with sockets...Also try Yahoo.com, I hear they have some pretty decent code to help with this.
Was This Post Helpful? 0
  • +
  • -

#3 syfran  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 83
  • View blog
  • Posts: 1,103
  • Joined: 12-July 09

Re: Program to collect data from a website

Posted 13 July 2009 - 10:32 PM

And if all that fails, look at the source code of the webpage and see if there is any easy way to get the information you need out of the source code a <div name="TreasureHere"> would be nice...
Was This Post Helpful? 0
  • +
  • -

#4 pbl  Icon User is offline

  • There is nothing you can't do with a JTable
  • member icon

Reputation: 8324
  • View blog
  • Posts: 31,857
  • Joined: 06-March 08

Re: Program to collect data from a website

Posted 14 July 2009 - 05:15 PM

[rules][/rules]
Was This Post Helpful? 0
  • +
  • -

#5 renegade334  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 9
  • Joined: 13-July 09

Re: Program to collect data from a website

Posted 14 July 2009 - 05:49 PM

@Dogstopper and syfran - Thank you. I shall examine this.

@pbl - Duly noted. Thank You.
Was This Post Helpful? 0
  • +
  • -

#6 renegade334  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 9
  • Joined: 13-July 09

Re: Program to collect data from a website

Posted 14 July 2009 - 05:56 PM

 
<WINNERS_LIST>
<span class="ubprotabletitle">

Current Winning Bids
</span>

<span class="ubprotablename">1. BM of Reno,NV</span><span class="ubprotablewin">$63</span>


<span class="ubprotablename">2. JS of SAN MARCOS,CA</span><span class="ubprotablewin">$62</span>

<span class="ubprotabletitle">Highest Unsuccessful Bids</span>

<span class="ubprotablename">3. DK of Orlando,FL</span><span class="ubprotablewin">$55</span>
<span class="ubprotablename">4. DK of Orlando,FL</span><span class="ubprotablewin">$55</span>


<span class="ubprotablename">5. MR of Van Nuys,CA</span><span class="ubprotablewin">$45</span>
<span class="ubprotablename">6. MR of Van Nuys,CA</span><span class="ubprotablewin">$45</span>


<span class="ubprotablename">7. YT of Campbell,CA</span><span class="ubprotablewin">$36</span>

</WINNERS_LIST>



This is how the source looks, for the part I will need to collect.

So would it work, if I were to write a program, downloading web pages I specify, reading them as text, looking for tags such as <winners-list> and storing the information into a xls file?

How do I use the FileReader method to read specific parts of a file?

Thanks
Was This Post Helpful? 0
  • +
  • -

#7 pbl  Icon User is offline

  • There is nothing you can't do with a JTable
  • member icon

Reputation: 8324
  • View blog
  • Posts: 31,857
  • Joined: 06-March 08

Re: Program to collect data from a website

Posted 14 July 2009 - 06:58 PM

How this that code related to Java programming ?
You are in a Java Programming forum
Nothing to do with Javascript or HTML
Was This Post Helpful? 0
  • +
  • -

#8 syfran  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 83
  • View blog
  • Posts: 1,103
  • Joined: 12-July 09

Re: Program to collect data from a website

Posted 14 July 2009 - 10:19 PM

Look into how to download a webpage to start with then you need to find a way to sort through the page you download. Try to get some code for that written.
Was This Post Helpful? 1

#9 renegade334  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 9
  • Joined: 13-July 09

Re: Program to collect data from a website

Posted 15 July 2009 - 06:12 AM

@pbl - I put that here because it is java script, and even though java and javascript are only superficially related, the key principles, names and naming conventions are the same. That's why I asked the question here.

I see you're a "mentor" here with over 7k posts. Are the majority of them like your last one above?

@syfran - I figured out a way to do that using grep once I've saved a webpage. Should be slightly simpler from here on. Thanks anyway!

This post has been edited by renegade334: 15 July 2009 - 06:11 AM

Was This Post Helpful? 0
  • +
  • -

#10 syfran  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 83
  • View blog
  • Posts: 1,103
  • Joined: 12-July 09

Re: Program to collect data from a website

Posted 15 July 2009 - 08:26 AM

View Postrenegade334, on 15 Jul, 2009 - 05:12 AM, said:

@pbl - I put that here because it is java script, and even though java and javascript are only superficially related, the key principles, names and naming conventions are the same. That's why I asked the question here.

I see you're a "mentor" here with over 7k posts. Are the majority of them like your last one above?

@syfran - I figured out a way to do that using grep once I've saved a webpage. Should be slightly simpler from here on. Thanks anyway!


I thought you were planning on implementing the idea in java. There is an entire forum on javascript.

This post has been edited by syfran: 15 July 2009 - 08:26 AM

Was This Post Helpful? 0
  • +
  • -

#11 renegade334  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 9
  • Joined: 13-July 09

Re: Program to collect data from a website

Posted 15 July 2009 - 08:27 AM

I am going to implement it in Java, not Jscript. Put that reply to pbl
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1