Problem with a robot posting to a web forum

Difficulties getting the results of a query, perhaps a problem with co

Page 1 of 1

0 Replies - 3574 Views - Last Post: 14 November 2006 - 01:15 PM Rate Topic: -----

#1 davellison  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 1
  • Joined: 14-November 06

Problem with a robot posting to a web forum

Posted 14 November 2006 - 01:15 PM

I am writing a robot, in Java, that will post to a web form and read the results. When I submit the web form however, it does not go through successfully and I get the web form back instead of the query results.

The attached file contains this code if you would like to download and compile it.

Thank you in advance, I have worked long and hard (unsuccesfully) to solve this problem by myself.

Overview of the Program I have written so far:
Court Reader Program
The purpose of this program is to extract the court records from the Maryland State website
First one must check a disclaimer, and a cookie with a session id is sent to you. ("")
The next page has three webforms, only one of which I care about
Then I attempt to Post to the second webform and read the response
I am unsuccessful at reading the results of the query, I only get back the page with the webform.

import java.util.*;

class CourtReaderTest
public static void main(String[] args) {
try {
First Start with the "disclaimer" page, it requires you to check a box and sends you a cookie with a session id.
I try to post this the easy way and simply append the disclaimer=Y to the end of the url. Apparently it is successful.
URL url = new URL("");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
//Find the cookie
String cookie = conn.getHeaderField("Set-Cookie");
//chop off the unecessary information after the semi-colon and save the session id information in the String "cookie"
int endCookieIndex = cookie.indexOf(";");
if(endCookieIndex >=0) cookie = cookie.substring(0,endCookieIndex);
if(cookie!=null)System.out.println("Cookie value " + cookie);

Open another connection to the page with the web form I actually care about (Unforetunately it drops me back to the disclaimer page if I try to jump to it directly and search)
I will attempt to set the HTTP header with the session id contained in the cookie previously extracted so the server remembers I have checked the disclaimer
URL url2 = new URL("");
//String url2String = ";" + cookie;
//URL url2 = new URL("");
HttpURLConnection conn2 = (HttpURLConnection) url2.openConnection();
//just confirming it will follow redirects, which it does
boolean followRedirects = conn2.getFollowRedirects();
System.out.println ("Follow Redriects? " + followRedirects);

OutputStream rawOutputStream = conn2.getOutputStream();
PrintWriter pw = new PrintWriter(rawOutputStream);
//I am writting the query contained in the webform. I copied this directly from a successful query sent by my browser, but there may be some Javascript processing that I have not acounted for
String queryUnencoded = "lastName=ELLISON&firstName=&middleName=&partyType=&site=00&courtSystem=B&countyName=&filingStart=&filingEnd=&filingDate=&company=N&action=Search";
String query = URLEncoder.encode(queryUnencoded, "UTF-8");
//post the query
attempt to read the response and print it to the console. Unforetuately I only read back the same page with the web form, and not the results I wanted.
This is the problem I want solved.
InputStream rawInStream = conn.getInputStream();
BufferedReader rdr = new BufferedReader(new InputStreamReader(rawInStream));
String line;
while ((line = rdr.readLine()) != null) {

} catch (MalformedURLException me) {
System.out.println("MalformedURLException: " + me);
} catch (IOException ioe) {
System.out.println("IOException: " + ioe);
} catch (Exception e) {
System.out.println("Exception: " + e);
}//end main method
}//end CourtReaderTest Class

Attached File(s)

Is This A Good Question/Topic? 0
  • +

Page 1 of 1