[Sergio Tapia]

Gah! I'm working on the rewrite of SharpDIC, now known as SharpDream, and I'd like to implement a method of FindAllMembers().

Now, since the site doesn't offer a nice way to ask for this I've done the following:

public IList<Member> FindAllMembers()
{
    var sourceLoader = new XmlMemberSource();
    var members = new List<Member>();

    using (var webClient = new WebClient())
    {
        for (int i = 1; i < 500; i++)
        {
            string response = webClient.DownloadString(String.Format("{0}{1}", "http://www.dreamincode.net/forums/xml.php?showuser=", i));
            var member = sourceLoader.LoadMember(response);

            if (member.Exists)
                members.Add(member);
        }
    }

    return members;
}

I don't want to be a dick and mess up your servers. :\

Is this OK? (light on your servers) or should you guys provide an exposed API for this purpose?

[skyhawk133]

Fine for now.

[Sergio Tapia]

Ok cool, thanks

[Shane Hudson]

It might be a good idea to have a database to store it in and sync? Rather than going through every member every time. Though I expect it would be a large database!

[Sergio Tapia]

Since this is a client library, do you think it would be a good idea to actually save a list of 450,000+ users? Honest question, is this something you'd expect in a library?

[SixOfEleven]

I don't think saving 450k records, or more, would be a good thing for a client application. The information already exists, connecting to an external data source is usually the better alternative. Unless you are told from the owner you're driving their bandwidth through the roof and to cease and desist. Then synching would be the way to go.

[Sergio Tapia]

yep

[Shane Hudson]

Good point.

It would be very handy if portions of the real database could be opened to us, but I am sure that would be far to large a security risk.

[Core]

Sergio, going through the entire user list to find one (especially if one is at the end) will take hours - done this myself to create a table of UserID - UserName associations. So using this method for a method call is not a better solution compared to keeping 450k+ members locally.

[Amrykid]

Core, on 24 March 2011 - 12:35 AM, said:

Sergio, going through the entire user list to find one (especially if one is at the end) will take hours - done this myself to create a table of UserID - UserName associations. So using this method for a method call is not a better solution compared to keeping 450k+ members locally.

Maybe DIC could host a xml file with every user (plus their id) in it. When it is first generated (sure, it would take hours) but after that, just append the new users as needed and cache/compress.

Better yet, a formated text file.

Now that file could be periodontally downloaded instead of going through every single person's profile to list all of them (possibly killing the server).

This post has been edited by Amrykid: 25 March 2011 - 05:25 PM

[Core]

That file will be +/- 120MB. Whenever someone will want to get data from it, he will have to download the entire file and that would be the least appropriate idea (esp. for mobile devices). A better solution would be exposing a service endpoint that would query a DB and return serialized data.