10 Replies - 12639 Views - Last Post: 25 March 2011 - 10:26 PM Rate Topic: -----

#1 Sergio Tapia  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1253
  • View blog
  • Posts: 4,168
  • Joined: 27-January 10

Is this kosher or am I raping your servers?

Posted 20 December 2010 - 06:43 PM

Gah! I'm working on the rewrite of SharpDIC, now known as SharpDream, and I'd like to implement a method of FindAllMembers().

Now, since the site doesn't offer a nice way to ask for this I've done the following:

public IList<Member> FindAllMembers()
{
    var sourceLoader = new XmlMemberSource();
    var members = new List<Member>();

    using (var webClient = new WebClient())
    {
        for (int i = 1; i < 500; i++)
        {
            string response = webClient.DownloadString(String.Format("{0}{1}", "http://www.dreamincode.net/forums/xml.php?showuser=", i));
            var member = sourceLoader.LoadMember(response);

            if (member.Exists)
                members.Add(member);
        }
    }

    return members;
}



I don't want to be a dick and mess up your servers. :\

Is this OK? (light on your servers) or should you guys provide an exposed API for this purpose?

Is This A Good Question/Topic? 0
  • +

Replies To: Is this kosher or am I raping your servers?

#2 skyhawk133  Icon User is offline

  • Head DIC Head
  • member icon

Reputation: 1877
  • View blog
  • Posts: 20,284
  • Joined: 17-March 01

Re: Is this kosher or am I raping your servers?

Posted 20 December 2010 - 06:46 PM

Fine for now.
Was This Post Helpful? 0
  • +
  • -

#3 Sergio Tapia  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1253
  • View blog
  • Posts: 4,168
  • Joined: 27-January 10

Re: Is this kosher or am I raping your servers?

Posted 20 December 2010 - 06:50 PM

Ok cool, thanks
Was This Post Helpful? 0
  • +
  • -

#4 Shane Hudson  Icon User is offline

  • D.I.C Technophile
  • member icon

Reputation: 343
  • View blog
  • Posts: 1,286
  • Joined: 06-December 09

Re: Is this kosher or am I raping your servers?

Posted 21 December 2010 - 06:18 AM

It might be a good idea to have a database to store it in and sync? Rather than going through every member every time. Though I expect it would be a large database!
Was This Post Helpful? 0
  • +
  • -

#5 Sergio Tapia  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1253
  • View blog
  • Posts: 4,168
  • Joined: 27-January 10

Re: Is this kosher or am I raping your servers?

Posted 21 December 2010 - 06:31 AM

Since this is a client library, do you think it would be a good idea to actually save a list of 450,000+ users? Honest question, is this something you'd expect in a library?
Was This Post Helpful? 0
  • +
  • -

#6 SixOfEleven  Icon User is offline

  • using Caffeine;
  • member icon

Reputation: 945
  • View blog
  • Posts: 6,342
  • Joined: 18-October 08

Re: Is this kosher or am I raping your servers?

Posted 21 December 2010 - 06:35 AM

I don't think saving 450k records, or more, would be a good thing for a client application. The information already exists, connecting to an external data source is usually the better alternative. Unless you are told from the owner you're driving their bandwidth through the roof and to cease and desist. Then synching would be the way to go.
Was This Post Helpful? 0
  • +
  • -

#7 Sergio Tapia  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1253
  • View blog
  • Posts: 4,168
  • Joined: 27-January 10

Re: Is this kosher or am I raping your servers?

Posted 21 December 2010 - 07:00 AM

yep
Was This Post Helpful? 0
  • +
  • -

#8 Shane Hudson  Icon User is offline

  • D.I.C Technophile
  • member icon

Reputation: 343
  • View blog
  • Posts: 1,286
  • Joined: 06-December 09

Re: Is this kosher or am I raping your servers?

Posted 21 December 2010 - 12:27 PM

Good point.

It would be very handy if portions of the real database could be opened to us, but I am sure that would be far to large a security risk.
Was This Post Helpful? 0
  • +
  • -

#9 Core  Icon User is offline

  • using System.Linq;
  • member icon

Reputation: 774
  • View blog
  • Posts: 5,097
  • Joined: 08-December 08

Re: Is this kosher or am I raping your servers?

Posted 23 March 2011 - 10:35 PM

Sergio, going through the entire user list to find one (especially if one is at the end) will take hours - done this myself to create a table of UserID - UserName associations. So using this method for a method call is not a better solution compared to keeping 450k+ members locally.
Was This Post Helpful? 0
  • +
  • -

#10 Amrykid  Icon User is offline

  • 4+1=Moo
  • member icon

Reputation: 148
  • View blog
  • Posts: 1,589
  • Joined: 16-December 08

Re: Is this kosher or am I raping your servers?

Posted 25 March 2011 - 05:14 PM

View PostCore, on 24 March 2011 - 12:35 AM, said:

Sergio, going through the entire user list to find one (especially if one is at the end) will take hours - done this myself to create a table of UserID - UserName associations. So using this method for a method call is not a better solution compared to keeping 450k+ members locally.

Maybe DIC could host a xml file with every user (plus their id) in it. When it is first generated (sure, it would take hours) but after that, just append the new users as needed and cache/compress.

Better yet, a formated text file.

Now that file could be periodontally downloaded instead of going through every single person's profile to list all of them (possibly killing the server).

This post has been edited by Amrykid: 25 March 2011 - 05:25 PM

Was This Post Helpful? 0
  • +
  • -

#11 Core  Icon User is offline

  • using System.Linq;
  • member icon

Reputation: 774
  • View blog
  • Posts: 5,097
  • Joined: 08-December 08

Re: Is this kosher or am I raping your servers?

Posted 25 March 2011 - 10:26 PM

That file will be +/- 120MB. Whenever someone will want to get data from it, he will have to download the entire file and that would be the least appropriate idea (esp. for mobile devices). A better solution would be exposing a service endpoint that would query a DB and return serialized data.
Was This Post Helpful? 1
  • +
  • -

Page 1 of 1