Totally new to even the thought of making a site.Wanting to use data from an existing site, to archive differently...
Page 1 of 1
11 Replies - 1703 Views - Last Post: 05 October 2010 - 05:53 PM
#1
Totally new to even the thought of making a site.
Posted 03 October 2010 - 08:52 AM
What I'm asking is, what exactly do I need physically? Obviously a server, and I'd prefer to use opensource software through out. So I'd probably use Ubuntu Server. And what programming languages in particular do I need to study aggressively (the most) as well as semi aggressive (Will need information from, but not so much as to rely on them).
Any help at all is appreciated. And any questions asked for more particular goals, I'll try to answer in a timely manner. Like I said I'm new, I dont know what specifics need to be answered to get a definitive answer.
Replies To: Totally new to even the thought of making a site.
#2
Re: Totally new to even the thought of making a site.
Posted 03 October 2010 - 09:37 AM
You will want to use a SQL Database for this. MySQL is commonly used, and is free. Also, there are a few common web development languages used today, including PHP, ColdFusion, and ASP.NET. I personally use PHP, but I'm sure the other two are just fine as well. Also, take a look at technologies like WAMP (Windows, Apache, MySQL and PHP) or LAMP (Linux, Apache, MySQL, and PHP).
#3
Re: Totally new to even the thought of making a site.
Posted 03 October 2010 - 08:50 PM
That's an important question with both technical and legal ramifications.
#4
Re: Totally new to even the thought of making a site.
Posted 04 October 2010 - 05:20 AM
#5
Re: Totally new to even the thought of making a site.
Posted 04 October 2010 - 12:52 PM
Hiring someone is not an option. I was trying to do a public service while learning. Outsourcing certain elements may be an option, but to just buy a project is self defeating. I'm not in a hurry to get it built. I just got the idea the other day. And even if Ohio puts a Facility inhabitants feature on their site, there are several States without I could re-gear the project for.
Thanks for input and suggestions. I appreciate it greatly.
#6
Re: Totally new to even the thought of making a site.
Posted 04 October 2010 - 01:12 PM
XHTML
CSS
Javascript
PHP
MySQL
I'd use the tutorials at w3schools and follow each tutorial up with reading through some examples on the site. Then follow that up with some practice projects of your own. At some point you'll need to be learning PHP and MySQL at the same time because working with a database is a major part of working with a server side language such as PHP.
*EDIT* and about the legality it sounds safe to me but you could always apply for an LLC and have that LLC pay you. That way if the site gets sued they can only take the company and not all the profit you've made.
This post has been edited by eTech1: 04 October 2010 - 01:13 PM
#7
Re: Totally new to even the thought of making a site.
Posted 04 October 2010 - 03:04 PM
Quote
But you haven't answered my question about how you have access to the data. It's important because it affects what programming work you have to do, and consequently, what languages you may learn. Do you have a raw dump of all the inmate data? Does the website provide a machine friendly way to access inmate data? This is the first issue to be tackled in this kind of project!
Also, to address what you physically need. I don't think you want to host the website yourself. You want to host it properly. That means server hosts who have proper and expensive hardware, with good backup and systems, have fat pipes to the internet, and are few hops away from the internet backbone.
For development purposes, you just need a regular computer. You don't need to install Linux. Exactly what software you install depends on what languages you pick.
I'd like to know the answer to the data source question, before I throw any input on what languages you need to study.
#8
Re: Totally new to even the thought of making a site.
Posted 04 October 2010 - 07:05 PM
Oler1s, on 04 October 2010 - 02:04 PM, said:
Quote
But you haven't answered my question about how you have access to the data. It's important because it affects what programming work you have to do, and consequently, what languages you may learn. Do you have a raw dump of all the inmate data? Does the website provide a machine friendly way to access inmate data? This is the first issue to be tackled in this kind of project!
Also, to address what you physically need. I don't think you want to host the website yourself. You want to host it properly. That means server hosts who have proper and expensive hardware, with good backup and systems, have fat pipes to the internet, and are few hops away from the internet backbone.
For development purposes, you just need a regular computer. You don't need to install Linux. Exactly what software you install depends on what languages you pick.
I'd like to know the answer to the data source question, before I throw any input on what languages you need to study.
The information would be gathered from http://www.drc.ohio....rch/Search.aspx .gov means government obviously and everything viewed is under public records. The only real problem I think I'd come into is "Bubba" being disappointed His picture is now in two places rather then one. The DRC site does not even have a terms of use or "condition of entry" pop up. Theres another website that's a publically traded Company I believe that uses data gathered from the National sex offenders registry and makes it a fancy GUI. I've never read about them catching flack.
The HTML is pretty setforth when I use firebug to look @ the code. Nothing seems "hidden". In fact they make the inmate # the picture file name so that would give me a route for back up.
I agree with you about using a hosting Company. If I had a generator and better access to a T3 connection I'd undoubtedly host it myself. I just kind of like the idea of having a server in the house to goof around with. Obviously I'd like to have 100% uptime. I'm practical thou and understand things will happen. And undoubtedly once the State Network Guys find out about my project they'll recode it so I'll have to some work.
#9
Re: Totally new to even the thought of making a site.
Posted 05 October 2010 - 08:43 AM
For this site to stay up to date you'll basically have to connect your search to theirs and parse the html file that their site spits out. It's possible but if they ever change the output of there search pages then you'd be down until you rewrite your parsing algorithm.
Another way would be do a search of each county, say once a day, and parse from there. Then your page would still be up to date every morning and you wouldn't be directly connecting to their site each time you get a visitor.
You mentioned images. My guess is public records only includes the information. I would highly recommend you ask first if you intend to use their images or even plan on hot-linking to them.
I would still say go with PHP and MySQL because they are going to be cheaper, there's more information about them on the web, they can get this job done and you get to choose your development environment.
This post has been edited by eTech1: 05 October 2010 - 08:45 AM
#10
Re: Totally new to even the thought of making a site.
Posted 05 October 2010 - 04:23 PM
Quote
Quote
You would want to write an automated scraper for this, of course. It's just a script or two that needs to be called periodically to update your own databases of inmate data. Python and Ruby (and Perl) work well for this need. You could also use C# or Java if you like, but this kind of high level scripting is done fastest in Python or Ruby. Pick your poison.
What you're going to do is create an HTTP client (like your browser). A form submission is nothing more than an HTTP request. You need to craft the appropriate request through code, and handle the response from the server. This response will be the HTML document.
So this is your starting point. To be able to actually write some code in one of those languages I mentioned. Pick one, and go through some basic language tutorials and practice a bit until you can write code.
#11
Re: Totally new to even the thought of making a site.
Posted 05 October 2010 - 05:25 PM
eTech1, on 05 October 2010 - 07:43 AM, said:
For this site to stay up to date you'll basically have to connect your search to theirs and parse the html file that their site spits out. It's possible but if they ever change the output of there search pages then you'd be down until you rewrite your parsing algorithm.
I was hoping I'd be able to just run the update in the background from like 03:00- whatevers a reasonable amount of time to fetch the DRC changes. I was thinking sort of like googles spiders. Where they dont archive every change individually, only modify.
eTech1, on 05 October 2010 - 07:43 AM, said:
I'm thinking I will play it safe now. Several people have mentioned it. And since I'm not in a huge hurry, I wouldnt mind the 3 week wait it takes the State to figure out where it placed it's pencil. I think the only thing that will piss me off about the project is if after I complete it, the web administrator strokes a couple keys and makes an even better version of the inmate tracker.
Oler1s, on 05 October 2010 - 03:23 PM, said:
Quote
Quote
You would want to write an automated scraper for this, of course. It's just a script or two that needs to be called periodically to update your own databases of inmate data. Python and Ruby (and Perl) work well for this need. You could also use C# or Java if you like, but this kind of high level scripting is done fastest in Python or Ruby. Pick your poison.
What you're going to do is create an HTTP client (like your browser). A form submission is nothing more than an HTTP request. You need to craft the appropriate request through code, and handle the response from the server. This response will be the HTML document.
So this is your starting point. To be able to actually write some code in one of those languages I mentioned. Pick one, and go through some basic language tutorials and practice a bit until you can write code.
If you wouldnt mind, how long do you think it would take a super proficient programmer, a moderately experienced programmer and a virgin programmer to complete such a program?
#12
Re: Totally new to even the thought of making a site.
Posted 05 October 2010 - 05:53 PM