12 Replies - 1692 Views - Last Post: 12 August 2012 - 11:54 PM

#1 carnivroar  Icon User is offline

  • D.I.C Regular

Reputation: 28
  • View blog
  • Posts: 382
  • Joined: 18-September 11

Random page from a website?

Posted 11 August 2012 - 05:29 PM

I would like to create a program to goes to a page, specifically http://www.classtab.org/, and returns a random link from the page.

Is this possible? And if so, what tools do I need for it?

Thanks.
Is This A Good Question/Topic? 0
  • +

Replies To: Random page from a website?

#2 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 8365
  • View blog
  • Posts: 31,080
  • Joined: 12-June 08

Re: Random page from a website?

Posted 11 August 2012 - 09:15 PM

A database of links and a bit of php to randomize the return from the database..
Was This Post Helpful? 0
  • +
  • -

#3 carnivroar  Icon User is offline

  • D.I.C Regular

Reputation: 28
  • View blog
  • Posts: 382
  • Joined: 18-September 11

Re: Random page from a website?

Posted 11 August 2012 - 09:21 PM

View Postmodi123_1, on 11 August 2012 - 09:15 PM, said:

A database of links and a bit of php to randomize the return from the database..


I've never done databases before but this would be a great place to get started. Do I have to extract the links manually or is there an automatic way of doing it? There's thousands of links on that site. I know there's a random Wikipedia article generator somewhere...
Was This Post Helpful? 0
  • +
  • -

#4 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 8365
  • View blog
  • Posts: 31,080
  • Joined: 12-June 08

Re: Random page from a website?

Posted 11 August 2012 - 09:24 PM

Yeah.. just like you would use a random number generate to snag a random spot in say an array it works the same.. but you would query a database and say "I want random row id X"....
Was This Post Helpful? 0
  • +
  • -

#5 Lemur  Icon User is offline

  • Pragmatism over Dogma
  • member icon



Reputation: 1335
  • View blog
  • Posts: 3,398
  • Joined: 28-November 09

Re: Random page from a website?

Posted 11 August 2012 - 10:15 PM

Use wget to get the page, grep for <a href and use sed to clean up a bit. You now have all the links, and it's using line numbers and a random generator.

Guitar player eh? I frequent the site.
Was This Post Helpful? 0
  • +
  • -

#6 carnivroar  Icon User is offline

  • D.I.C Regular

Reputation: 28
  • View blog
  • Posts: 382
  • Joined: 18-September 11

Re: Random page from a website?

Posted 11 August 2012 - 10:19 PM

View PostLemur, on 11 August 2012 - 10:15 PM, said:

Use wget to get the page, grep for <a href and use sed to clean up a bit. You now have all the links, and it's using line numbers and a random generator.

Guitar player eh? I frequent the site.


I already did it in Python, I managed to get the list of text files easily. Now I need to convert it into Javascript.

import random

list = open("list.txt", "r").read().split("\n")

print("http://www.classtab.org/" + list[random.randint(0, len(list))])



This doesn't seem to work:

function load(){
	var fs = require("fs");
	var array = fs.readFileSync("list.txt").toString().split("\n");
	window.alert(array[Math.floor(Math.random()*array.length())])
}



The problem is with the require("fs") thing. I found that piece of code on google.

This post has been edited by carnivroar: 11 August 2012 - 10:19 PM

Was This Post Helpful? 0
  • +
  • -

#7 carnivroar  Icon User is offline

  • D.I.C Regular

Reputation: 28
  • View blog
  • Posts: 382
  • Joined: 18-September 11

Re: Random page from a website?

Posted 11 August 2012 - 10:45 PM

http://art.lehman.ed...va/randtab.html

Bingo!

But I'm not sure if it's a good idea to have a big ass array in the code.

This post has been edited by carnivroar: 12 August 2012 - 12:07 AM

Was This Post Helpful? 0
  • +
  • -

#8 atraub  Icon User is offline

  • Pythoneer
  • member icon

Reputation: 756
  • View blog
  • Posts: 1,990
  • Joined: 23-December 08

Re: Random page from a website?

Posted 12 August 2012 - 11:31 AM

In Python, you could probably use a generator rather than a list, not sure what you'd do in JS though.

This post has been edited by atraub: 12 August 2012 - 11:37 AM

Was This Post Helpful? 0
  • +
  • -

#9 carnivroar  Icon User is offline

  • D.I.C Regular

Reputation: 28
  • View blog
  • Posts: 382
  • Joined: 18-September 11

Re: Random page from a website?

Posted 12 August 2012 - 03:08 PM

View Postatraub, on 12 August 2012 - 11:31 AM, said:

In Python, you could probably use a generator rather than a list, not sure what you'd do in JS though.


Can you embed python code on a website? I looked it up but it seems so complicated. I think there has to be a better way than storing all the text file names in an array.
Was This Post Helpful? 0
  • +
  • -

#10 alexr1090  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 38
  • View blog
  • Posts: 108
  • Joined: 08-May 11

Re: Random page from a website?

Posted 12 August 2012 - 06:15 PM

View Postcarnivroar, on 11 August 2012 - 07:29 PM, said:

I would like to create a program to goes to a page, specifically http://www.classtab.org/, and returns a random link from the page.

Is this possible? And if so, what tools do I need for it?

Thanks.


Yep it's possible. You need to import urllib2, random, and an html parser (I like BeautifulSoup).
Was This Post Helpful? 0
  • +
  • -

#11 alexr1090  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 38
  • View blog
  • Posts: 108
  • Joined: 08-May 11

Re: Random page from a website?

Posted 12 August 2012 - 06:43 PM

import urllib2, random
from bs4 import BeautifulSoup

#get html and turn it into soup
html = urllib2.urlopen('http://www.classtab.org/')
soup = BeautfiulSoup(html, "lxml")

links = soup.find_all('a')
count = len(links)

#now create a random number between 1 and count
#do that lol I forgot how random works!

html2 = urllib2.urlopen('http://www.classtab.org/'+links[randomnumber].get('href'))
soup2 = BeautifulSoup(html2, "lxml")
print soup2.get_text()

###haven't tested this. Hope it works and helps! Good luck.


Was This Post Helpful? 1
  • +
  • -

#12 atraub  Icon User is offline

  • Pythoneer
  • member icon

Reputation: 756
  • View blog
  • Posts: 1,990
  • Joined: 23-December 08

Re: Random page from a website?

Posted 12 August 2012 - 07:13 PM

Web browsers can't run python code, but web servers can. You can embed python-like code in a page using Django (and probably other web frameworks as well).
Was This Post Helpful? 0
  • +
  • -

#13 carnivroar  Icon User is offline

  • D.I.C Regular

Reputation: 28
  • View blog
  • Posts: 382
  • Joined: 18-September 11

Re: Random page from a website?

Posted 12 August 2012 - 11:54 PM

Wait a minute - why was this moved to Python? I posted this on Web Development. lol.

And I need to implement a better script because it's very slow on Firefox (FF cannot handle Javascript very well).

This post has been edited by carnivroar: 12 August 2012 - 11:54 PM

Was This Post Helpful? 0
  • +
  • -

Page 1 of 1