5 Replies - 382 Views - Last Post: 04 July 2012 - 12:49 PM Rate Topic: -----

#1 jake_123  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 3
  • Joined: 03-July 12

Fastest code/way to get values from table into text

Posted 03 July 2012 - 02:21 PM

First of all apologies to the expert OO programmers - I only know Matlab. :(


Trying to find the fastest code to extract only the last 2 colums (all rows) from an html table. I've tried the below which works but I don;t want it to write to cdm window/console. It dissapears anyway. I want for example to use the values in another class or pass them on to something else. Also not sure what libraries to import always so plese show your libraries too.


[using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.Net;

namespace ConsoleApplication3
{
class WebFetch
{
static void Main(string[] args)
{
StringBuilder sb = new StringBuilder(); //Used to StringBuilder entire input

byte[] buf = new byte[8192];

HttpWebRequest request = (HttpWebRequest) //Prepare the Webpage we will be asking for
WebRequest.Create("http://www.eia.gov/dnav/pet/pet_sum_sndw_dcus_nus_w.htm");

HttpWebResponse response = (HttpWebResponse)
request.GetResponse();

Stream resStream = response.GetResponseStream();

string tempString = null;
int count = 0;

do
{
count = resStream.Read(buf, 0, buf.Length);

if (count != 0)
{
tempString = Encoding.ASCII.GetString(buf, 0, count);

sb.Append(tempString);
}
}
while (count > 0);

Console.WriteLine(sb.ToString());

}
}
}
]


I;ve also tried this but getting errors:

[using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.Net;

class Program2
{
static void Main(string[] args)
{
public String GetTableSource()
{
//Get the Search Address
String address = @"http://www.eia.gov/dnav/pet/pet_sum_sndw_dcus_nus_w.htm";
//Create an Instance of the WebClient Class
System.Net.WebClient client = new System.Net.WebClient();
//Download the Source
String reply = client.DownloadString(address);
//Will remove everything up to the table, Assuming there is only one Table on the Page
reply = reply.Remove(0, reply.IndexOf("<table"));
//Remove everything after </table>
reply = reply.Remove(reply.IndexOf("</table>") + 8);
//Return the Source
return reply;
}

}
}]

Is This A Good Question/Topic? 0
  • +

Replies To: Fastest code/way to get values from table into text

#2 Skydiver  Icon User is offline

  • Code herder
  • member icon

Reputation: 1942
  • View blog
  • Posts: 5,778
  • Joined: 05-May 12

Re: Fastest code/way to get values from table into text

Posted 03 July 2012 - 02:48 PM

In general, page scraping sucks. Can't you download the spreadsheet instead and pull the data using Excel's automation interface? http://www.eia.gov/d..._DCUS_NUS_W.xls
Was This Post Helpful? 0
  • +
  • -

#3 jake_123  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 3
  • Joined: 03-July 12

Re: Fastest code/way to get values from table into text

Posted 04 July 2012 - 11:46 AM

Thanks Skydriver,

Sure I can which is what I used, however it takes a few hundred milliseconds longer, and its all about trading on milliseconnds algorithms. The C# code will give me the string numbers faster by just looking at the html.
Was This Post Helpful? 0
  • +
  • -

#4 Skydiver  Icon User is offline

  • Code herder
  • member icon

Reputation: 1942
  • View blog
  • Posts: 5,778
  • Joined: 05-May 12

Re: Fastest code/way to get values from table into text

Posted 04 July 2012 - 11:55 AM

But that data that you are downloading is at least a week old, what would a few more milliseconds matter?
Was This Post Helpful? 0
  • +
  • -

#5 jake_123  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 3
  • Joined: 03-July 12

Re: Fastest code/way to get values from table into text

Posted 04 July 2012 - 12:07 PM

Yes its a week old now, but not when new data gets released weekly.
Was This Post Helpful? 0
  • +
  • -

#6 Skydiver  Icon User is offline

  • Code herder
  • member icon

Reputation: 1942
  • View blog
  • Posts: 5,778
  • Joined: 05-May 12

Re: Fastest code/way to get values from table into text

Posted 04 July 2012 - 12:49 PM

So you will be actively polling the iea.gov site every 500ms waiting for an update? (I didn't see an RSS feed for that page). That is a great way to get banned from a website.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1