5 Replies - 1782 Views - Last Post: 06 November 2012 - 01:55 PM Rate Topic: -----

#1 tpimtts  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 6
  • Joined: 17-October 12

Confused! Help! Librarys! Programming concepts?

Posted 20 October 2012 - 02:24 PM

Hi everyone;

This is a really great forum and there's lots of useful information here, so thanks to everyone for your advice and input.

I have a somewhat abstract question and I'm not entirely sure what my question is (which is part of the problem).. so please bare with me.

Some background information regarding me and my ambitions and goals....

I'm into networking and I'm a general computing enthusiast -- I understand scripting and indeed write lots of general scripts in bash for generic systems administration and networking tasks.

Currently, I have been reading up and studying python as my next step/progression from bash, as I'm interested in getting into the network programming aspect. I have been messing around with sockets, writing simple client/server applications in python and been progressing in a steady upward curve (in my opinion).

What I want to understand is more of the concepts and idea's behind writing larger programs... I want to understand I mean... Sorry, I find it hard to express what exactly I'm trying to get across.. I guess, I want to understand the concepts behind standard librarys, as in -- python has a general sort of "scripting" aspect but also, the object oriented programming style -- and it seems that if we want to make complex programs, do we make use of the python standard library.. as in for example tkinter, (if I understand correctly) it is a library of functions specifically for developing a GUI?? I just don't understand how to learn how the functions work in order to use them... I mean, I've seen tutorials but I don't understand how the writers of said tutorials learnt how to use these libraries in the first place? And how do we know what libraries are out there? Sorry if my questions are vague and not to the point, I just feel like I've hit an understanding barrier, and I feel confused. In a lot of tutorials and books I read, the imports are just imported without explaination as to where they have come from, who developed them, how people are expected to know what functions/classes do and what arguments and members they have etc.

If I have got this right, and please correct me if I'm wrong, utilising programming languages, starts in a sort of step by step process? As in... the newbie like myself, should try to understand the basic standard language.. and once they are competant in that, they can move to the next step of making use of different libraries like if we wanted a graphical 3d game, we can make use of the SDL library in C++? have I got that right...? But how do we know how to use the SDL library?

I will apologise again for a very obscure posting, but I feel just weirdly confused with libraries and dont know where to start understanding them or how to implement them.

Thanks in advance for any help.

Is This A Good Question/Topic? 1
  • +

Replies To: Confused! Help! Librarys! Programming concepts?

#2 Tayacan  Icon User is offline

  • D.I.C Regular
  • member icon

Reputation: 145
  • View blog
  • Posts: 275
  • Joined: 18-January 11

Re: Confused! Help! Librarys! Programming concepts?

Posted 21 October 2012 - 07:14 AM

Okay, so I think you want to know how the people who write the tutorials know stuff. Well, usually, the people who wrote a library (or language) in the first place, will write detailed documentation and perhaps even tutorials. Like, for Python, you can always look things up here. For example, try searching for Tkinter on that site.

At some point, though, someone will think they can explain it better - and maybe they can - and so they write another tutorial.

Another maybe-question that I'm reading out of your post is "How do I progress from newbie to master?". You're absolutely right that the conventional (and probably best) path is to learn the basics of the language - the syntax, the way to think about problems - and then start building more complex stuff. Again, for learning SDL, you'd look up SDL documentation and tutorials.
Was This Post Helpful? 1
  • +
  • -

#3 Python_4_President  Icon User is offline

  • D.I.C Regular

Reputation: 53
  • View blog
  • Posts: 321
  • Joined: 13-August 11

Re: Confused! Help! Librarys! Programming concepts?

Posted 23 October 2012 - 06:18 PM

I am by no means a master of anything, nor am I that great at explaining things, but maybe this will help. It's how I've learned to understand the concept of a "library".

As you may be well aware, when you're posed with the notion of "wanting to do X", in order to "do X", you must first do this, then do that, but that is kind of ugly, so you do this and it looks better, now you can do this over here and things start working out, but then you realize that "What you've done thus far" is hard to understand, so you sit still and ponder what you just told your computer to do, and perhaps, if you're lucky, a book falls from your shelf and hits you in the face, and at that moment, you realize what you've been trying to do all along is ttthhiisss.



Since the OS module in python is absolutely fantastic, let's take a peak at how you might create os.walk if os.walk didn't exist, or you didn't even know os.walk exists, but you did know the os module exists and knew a couple of functions exposed by it.

NOTE: I am not an OS developer, thus I could not explain to you how to identify what something is, or is not, just by looking at a bunch of 1s and 0s. Guido Van Rossum is a freakin' genius, though, so if you're really curious, explore the OS module source code.


Step 1: Origins...
#We might try, say...
import os


def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir




Hey, cool, we printed a string. I guess the thing we really want to know is... what's in this topleveldir? That sounds like we're asking for a list of things... in that topleveldir.

Step2: Exploring further

after you import os, you can do help(os) and get its documentation. At the top of the page you'll see, among other things, os.path, os.name, os.curdir, os.pardir, os.sep, os.extsep, os.altsep, os.pathsep, os.linesep, os.defpath, os.devnull.

Huh.. it says os.path is a module. I thought os was a module? Maybe that means modules can contain modules?

is os.name a module? Yes...

is a library a module? Yes...

is a module a library? Yes...

is a library a collection of easier-than-the-alternative tools that do things you need to do "in order to do X"? Yes...

DESCRIPTION
This module provides access to operating system functionality that is
standardized by the C Standard and the POSIX standard (a thinly
disguised Unix interface). Refer to the library manual and
corresponding Unix manual entries for more information on calls.


Well that's just crazy, but we're off on a yak shaving adventure. What we really wanted was a list of directory contents! Keyword there being list, since this is Python, after all, and lists are our 20 dollar bills. Who gets excited at a 20 dollar bill? I do!

We COULD write a function that lists the contents of a directory (and that is, indeed, what library builders do), but we are not that ninja.

refer to the output of help(os) again
Look for the first thing that says list.

    listdir(...)
        listdir(path) -> list_of_strings

        Return a list containing the names of the entries in the directory.

            path: path of directory to list

        The list is in arbitrary order.  It does not include the specia
        entries '.' and '..' even if they are present in the directory.



Huh... that sounds right up our alley.

Let's go back to our program to walk a directory, dir by dir, file by file..

#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    for thing in os.listdir(topleveldir):
        print "Hmm, here's a thing. ", thing
    
    
if __name__ == "__main__":
    walkThisDir("C:\\")




Hey, neat! Check this out! Ok.. Python27 is a directory, I know that for sure, because that's where I installed Python! $Recycle.Bin, wtf is that? I thought Recycle Bin is a directory? Weird...
.rnd? No telling, but in unix terms, it could be a file or a directory, who knows what it is in Windows.

Config.SYS MUST be a file, because it has an extension... right?

Wouldn't it be awesome if we had a tool that would check a thing and tell us what it is?


OK, walking through:  C:\
Hmm, here's a thing.  $Recycle.Bin
Hmm, here's a thing.  .rnd
Hmm, here's a thing.  Administrative Tools
Hmm, here's a thing.  AUTOEXEC.BAT
Hmm, here's a thing.  Boot
Hmm, here's a thing.  Boot.BAK
Hmm, here's a thing.  Boot.ini.saved
Hmm, here's a thing.  bootmgr
Hmm, here's a thing.  BOOTSECT.BAK
Hmm, here's a thing.  CONFIG.SYS
Hmm, here's a thing.  Crash
Hmm, here's a thing.  DirectX
Hmm, here's a thing.  Documents and Settings
Hmm, here's a thing.  downloads
Hmm, here's a thing.  Electronic Arts
Hmm, here's a thing.  Fraps
Hmm, here's a thing.  hiberfil.sys
Hmm, here's a thing.  Intel
Hmm, here's a thing.  IO.SYS
Hmm, here's a thing.  MSDOS.SYS
Hmm, here's a thing.  NTDETECT.COM
Hmm, here's a thing.  ntldr
Hmm, here's a thing.  ntuser.ini
Hmm, here's a thing.  NVIDIA
Hmm, here's a thing.  pagefile.sys
Hmm, here's a thing.  PerfLogs
Hmm, here's a thing.  Program Files
Hmm, here's a thing.  Program Files (x86)
Hmm, here's a thing.  ProgramData
Hmm, here's a thing.  Python27
Hmm, here's a thing.  Recovery
Hmm, here's a thing.  RECYCLER
Hmm, here's a thing.  System Volume Information
Hmm, here's a thing.  tmp
Hmm, here's a thing.  user.js
Hmm, here's a thing.  Users
Hmm, here's a thing.  Windows
Hmm, here's a thing.  Windows Messenger.lnk
Hmm, here's a thing.  Windows.old
Hmm, here's a thing.  Windows.old.000
Hmm, here's a thing.  WINXP
Hmm, here's a thing.  _appdata_





let's refer to the help(os) again, and see if there's anything that can help us out..

I see:
    open(...)
        open(filename, flag [, mode=0777]) -> fd

        Open a file (for low level IO).



It says it opens a file.. I guess if a directory is not the same as a file, trying to open it might give us an indicator (read: exception/exploding computer), and then we'd know, right? Seems really ineffective to do that, though. The smart people probably would laugh at such a crazy notion.

Same for remove removedirs and rmdir. Obviously that's a bad idea, but we can see times when doing something like that would be useful, right?

I need to remove a file -> os.remove("path/to/file")
I need to remove a dir -> os.rmdir("path/to/dir")
I wish I could remove an entire hierarchy of directories.. -> os.removedirs("path/to/topleveldir")

Neat! Tools are awesome! Continuing on through os you might come across stat.

Stat sounds nifty, what does it do?
    stat(...)
        stat(path) -> stat result

        Perform a stat system call on the given path.




Let's modify our program again..

#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    for thing in os.listdir(topleveldir):
        print "Hmm, here's a thing. ", thing
        print "stating it yields: ", os.stat(thing)
    
    
if __name__ == "__main__":
    walkThisDir("C:\\")




WAIT A MINUTE!
OK, walking through:  C:\
Hmm, here's a thing.  $Recycle.Bin
stating it yields: 
Traceback (most recent call last):
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 12, in <module>
    walkThisDir("C:\\")
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 8, in walkThisDir
    print "stating it yields: ", os.stat(thing)
WindowsError: [Error 2] The system cannot find the file specified: '$Recycle.Bin'



I found the thing, how come I can't stat it? Ooohh... right... paths are important..

Well, how could I assemble a path based on what I know about where I am?

I could try: topleveldir+thing Maybe that will work..

YEP! What the heck is all of that, though?
Hmm, here's a thing.  WINXP
stating it yields:  nt.stat_result(st_mode=16895, st_ino=0L, st_dev=0, st_nlink=0, st_uid=0, st_gid=0, st_size=28672L, st_atime=1337099325L, st_mtime=1337099324L, st_ctime=1336951988L)



Maybe stat isn't so helpful...

Let's refer back to the help(os) output.. If you continue on (please ignore the entry for walk, as it may spoil you, or just confuse you) you'll see that nothing in os seems to let you know what something is or is not.. It seems like that sort of utility would be part of an OS module though, right?

Flip back to the top of help(os)


Well, the curdir isn't important to us yet, the pardir doesn't even make sense, really because the curdir is the pardir so far, sep doesn't matter yet, and all that other stuff seems irrelevant.

HOW DO I TELL WHAT SOMETHING IS? Well, WHERE IS SOMETHING? Path...

help(os.path)


Oh boy! New tools!

    abspath(path)
        Return the absolute version of a path.

    basename(p)
        Returns the final component of a pathname

    commonprefix(m)
        Given a list of pathnames, returns the longest common leading component

    exists(path)
        Test whether a path exists.  Returns False for broken symbolic links

    expanduser(path)
        Expand ~ and ~user constructs.

        If user or $HOME is unknown, do nothing.




Huh.. Well, abspath would obviously be helpful at some point, but not for my purposes now.
Basename is probably pretty cool, but can't see a use for that either, yet.
commonprefix doesn't really make sense now.
dirname seems pretty cool, but no idea why I might want it now.
exists.. hey.. that's nifty. does os.path.exists("c:\\the\\moon")?

>>> os.path.exists("C:\\the\\moon")
False



WHAT? Lies...

Hey, wait a minute... Maybe this is a time for that makedirs thing..
>>> os.makedirs("C:\\the\\moon")
>>> os.path.exists("C:\\the\\moon")
True



Cool! I can tell if something exists or not, but makedirs says it makes dirs. moon must be a dir, but how should I know?

help(os.path)

scroll past the getting accessed, modified, and created time functions. Only nerds need those, and probably the only person in the world who cares about expanding a user is Linus Torvalds.

Anyone that cares about expanding a var is probably from the future, or writes a lot of BASH code.

wait a minute.. is.. do you know what the definition of is is? is is your ticket telling what's up. In most libraries, is is gold. get is platinum, and set is a brand new rolex. Keep that in mind.

ISABS??? meh
ISDIR!
ISFILE!
ISLINK? OH, SYM-BO-LIC LINK!

That sounds pretty cool.

Let's revise the program and try out some new stuff!
#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    for thing in os.listdir(topleveldir):
        print "Hmm, here's a thing. ", thing
        print "Is it a file? ", os.path.isfile(topleveldir+thing)
        print "Is it a dir? ", os.path.isdir(topleveldir+thing)
        print "Is it an absolute path? ", os.path.isabs(topleveldir+thing)
        print "Is it a symbolic link? ", os.path.islink(topleveldir+thing)
    
    
if __name__ == "__main__":
    walkThisDir("C:\\")




Hmm, here's a thing.  $Recycle.Bin
Is it a file?  False
Is it a dir?  True
Is it an absolute path?  True
Is it a symbolic link?  False
Hmm, here's a thing.  .rnd
Is it a file?  True
Is it a dir?  False
Is it an absolute path?  True
Is it a symbolic link?  False
Hmm, here's a thing.  Administrative Tools
Is it a file?  False
Is it a dir?  True
Is it an absolute path?  True
Is it a symbolic link?  False
Hmm, here's a thing.  AUTOEXEC.BAT
Is it a file?  True
Is it a dir?  False
Is it an absolute path?  True
Is it a symbolic link?  False




Huh.. $Recycle.Bin is a directory afterall..
.rnd turned out to be a file. NEAT!

Let's revise the code to keep track of directories and files.


#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    dirs = []
    files = []
    for thing in os.listdir(topleveldir):
        if os.path.isfile(topleveldir+thing):
            files.append(thing)
        elif os.path.isdir(topleveldir+thing):
            dirs.append(thing)
    print "Directories here: ", str(dirs)
    print "Files here: ", str(files)
        

    
    
if __name__ == "__main__":
    walkThisDir("C:\\")




OK, walking through:  C:\
Directories here:  ['$Recycle.Bin', 'Administrative Tools', 'Boot', 'Crash', 'DirectX', 'Documents and Settings', 'downloads', 'Electronic Arts', 'Fraps', 'Intel', 'NVIDIA', 'PerfLogs', 'Program Files', 'Program Files (x86)', 'ProgramData', 'Python27', 'Recovery', 'RECYCLER', 'System Volume Information', 'the', 'tmp', 'Users', 'Windows', 'Windows.old', 'Windows.old.000', 'WINXP', '_appdata_']

Files here:  ['.rnd', 'AUTOEXEC.BAT', 'Boot.BAK', 'Boot.ini.saved', 'bootmgr', 'BOOTSECT.BAK', 'CONFIG.SYS', 'hiberfil.sys', 'IO.SYS', 'MSDOS.SYS', 'NTDETECT.COM', 'ntldr', 'ntuser.ini', 'pagefile.sys', 'user.js', 'Windows Messenger.lnk']





COOL!

Well, what about stuff in other directories? I guess recursion is a good strategy here... A nice depth-first-search algorithm...

#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    dirs = []
    files = []
    for thing in os.listdir(topleveldir):
        if os.path.isfile(topleveldir+thing):
            files.append(thing)
        elif os.path.isdir(topleveldir+thing):
            dirs.append(thing)
    print "Directories here: ", str(dirs)
    print "Files here: ", str(files)
    for d in dirs:
        walkThisDir(topleveldir+d)
        

    
    
if __name__ == "__main__":
    walkThisDir("C:\\")




Huh.. It looks like it tried to work, but didn't.. Why not?


OK, walking through:  C:\Boot
Directories here:  []
Files here:  []
OK, walking through:  C:\Crash
Directories here:  []
Files here:  []
OK, walking through:  C:\DirectX
Directories here:  []
Files here:  []
OK, walking through:  C:\Documents and Settings
Traceback (most recent call last):
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 22, in <module>
    walkThisDir("C:\\")
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 16, in walkThisDir
    walkThisDir(topleveldir+d)
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 8, in walkThisDir
    for thing in os.listdir(topleveldir):
WindowsError: [Error 5] Access is denied: 'C:\\Documents and Settings/*.*'




Maybe the paths are weird?

#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    dirs = []
    files = []
    for thing in os.listdir(topleveldir):
        print "Where is this thing? ", topleveldir+thing
        if os.path.isfile(topleveldir+thing):
            files.append(thing)
        elif os.path.isdir(topleveldir+thing):
            dirs.append(thing)
    print "Directories here: ", str(dirs)
    print "Files here: ", str(files)
    for d in dirs:
        print "Where are we? ", topleveldir+d
        walkThisDir(topleveldir+d)
        

    
    
if __name__ == "__main__":
    walkThisDir("C:\\")




Where are we?  C:\DirectX
OK, walking through:  C:\DirectX
Where is this thing?  C:\DirectXApr2005_d3dx9_25_x64.cab
Where is this thing?  C:\DirectXApr2005_d3dx9_25_x86.cab
Where is this thing?  C:\DirectXApr2006_d3dx9_30_x64.cab
Where is this thing?  C:\DirectXApr2006_d3dx9_30_x86.cab
Where is this thing?  C:\DirectXApr2006_MDX1_x86.cab
Where is this thing?  C:\DirectXOCT2006_XACT_x86.cab
Directories here:  []
Files here:  []
Where are we?  C:\Documents and Settings
OK, walking through:  C:\Documents and Settings
Traceback (most recent call last):
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 24, in <module>
    walkThisDir("C:\\")
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 18, in walkThisDir
    walkThisDir(topleveldir+d)
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 8, in walkThisDir
    for thing in os.listdir(topleveldir):
WindowsError: [Error 5] Access is denied: 'C:\\Documents and Settings/*.*'



Well... i don't know about you, but C:\DirectXOCT2006_d3dx9_31_x86.cab seems to be missing something... A PATH SEPARATOR!!!

Surely topleveldir + thing does the right stuff right? WRONG, it works because Windows is weird, but only in the topmost level, since we included the separator in the one and only parameter to walkThisDir. Everything else is left to fend for itself.

Wouldn't it be nice to have a tool to work that magic out for us?

What are we doing when we say things like, "C:\this\path\is\here"?
Are we, say, JOINing things in a tree of things? with separators? Yeah, I can live with that.
Let's check os.path for something that has the word join in it.

  join(a, *p)
      Join two or more pathname components, inserting "\" as needed.
      If any component is an absolute path, all previous path components
      will be discarded.




Let's change it up a bit...



Hey, that sounds like just what we want! Let's revise the program.
[code]
#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    dirs = []
    files = []
    for thing in os.listdir(topleveldir):
        if os.path.isfile(os.path.join(topleveldir, thing)):
            print "added file ", thing
            files.append(thing)
        elif os.path.isdir(os.path.join(topleveldir, thing)):
            print "added dir ", thing
            dirs.append(thing)
    print "Directories here: ", str(dirs)
    print "Files here: ", str(files)
    for d in dirs:
        print "Where are we? ", os.path.join(topleveldir, d)
        walkThisDir(os.path.join(topleveldir, d))
        

    
    
if __name__ == "__main__":
    walkThisDir("C:\\")



added file  OCT2006_d3dx9_31_x64.cab
added file  OCT2006_d3dx9_31_x86.cab
added file  OCT2006_XACT_x64.cab
added file  OCT2006_XACT_x86.cab
Directories here:  []
Files here:  ['Apr2005_d3dx9_25_x64.cab', 'Apr2005_d3dx9_25_x86.cab', 'Apr2006_d3dx9_30_x64.cab', 'Apr2006_d3dx9_30_x86.cab', 'Apr2006_MDX1_x86.cab', 'Apr2006_MDX1_x86_Archive.cab', 'Apr2006_XACT_x64.cab', 'Apr2006_XACT_x86.cab', 'Apr2006_xinput_x64.cab', 'Apr2006_xinput_x86.cab', 'APR2007_d3dx10_33_x64.cab', 'APR2007_d3dx10_33_x86.cab', 'APR2007_d3dx9_33_x64.cab', 'APR2007_d3dx9_33_x86.cab', 'APR2007_XACT_x64.cab', 'APR2007_XACT_x86.cab', 'APR2007_xinput_x64.cab', 'APR2007_xinput_x86.cab', 'Aug2005_d3dx9_27_x64.cab', 'Aug2005_d3dx9_27_x86.cab', 'AUG2006_XACT_x64.cab', 'AUG2006_XACT_x86.cab', 'AUG2006_xinput_x64.cab', 'AUG2006_xinput_x86.cab', 'AUG2007_d3dx10_35_x64.cab', 'AUG2007_d3dx10_35_x86.cab', 'AUG2007_d3dx9_35_x64.cab', 'AUG2007_d3dx9_35_x86.cab', 'AUG2007_XACT_x64.cab', 'AUG2007_XACT_x86.cab', 'Aug2008_d3dx10_39_x64.cab', 'Aug2008_d3dx10_39_x86.cab', 'Aug2008_d3dx9_39_x64.cab', 'Aug2008_d3dx9_39_x86.cab', 'Aug2008_XACT_x64.cab', 'Aug2008_XACT_x86.cab', 'Aug2008_XAudio_x64.cab', 'Aug2008_XAudio_x86.cab', 'Aug2009_D3DCompiler_42_x64.cab', 'Aug2009_D3DCompiler_42_x86.cab', 'Aug2009_d3dcsx_42_x64.cab', 'Aug2009_d3dcsx_42_x86.cab', 'Aug2009_d3dx10_42_x64.cab', 'Aug2009_d3dx10_42_x86.cab', 'Aug2009_d3dx11_42_x64.cab', 'Aug2009_d3dx11_42_x86.cab', 'Aug2009_d3dx9_42_x64.cab', 'Aug2009_d3dx9_42_x86.cab', 'Aug2009_XACT_x64.cab', 'Aug2009_XACT_x86.cab', 'Aug2009_XAudio_x64.cab', 'Aug2009_XAudio_x86.cab', 'Dec2005_d3dx9_28_x64.cab', 'Dec2005_d3dx9_28_x86.cab', 'DEC2006_d3dx10_00_x64.cab', 'DEC2006_d3dx10_00_x86.cab', 'DEC2006_d3dx9_32_x64.cab', 'DEC2006_d3dx9_32_x86.cab', 'DEC2006_XACT_x64.cab', 'DEC2006_XACT_x86.cab', 'DSETUP.dll', 'dsetup32.dll', 'dxdllreg_x86.cab', 'DXSETUP.exe', 'dxupdate.cab', 'Feb2005_d3dx9_24_x64.cab', 'Feb2005_d3dx9_24_x86.cab', 'Feb2006_d3dx9_29_x64.cab', 'Feb2006_d3dx9_29_x86.cab', 'Feb2006_XACT_x64.cab', 'Feb2006_XACT_x86.cab', 'FEB2007_XACT_x64.cab', 'FEB2007_XACT_x86.cab', 'Feb2010_X3DAudio_x64.cab', 'Feb2010_X3DAudio_x86.cab', 'Feb2010_XACT_x64.cab', 'Feb2010_XACT_x86.cab', 'Feb2010_XAudio_x64.cab', 'Feb2010_XAudio_x86.cab', 'Jun2005_d3dx9_26_x64.cab', 'Jun2005_d3dx9_26_x86.cab', 'JUN2006_XACT_x64.cab', 'JUN2006_XACT_x86.cab', 'JUN2007_d3dx10_34_x64.cab', 'JUN2007_d3dx10_34_x86.cab', 'JUN2007_d3dx9_34_x64.cab', 'JUN2007_d3dx9_34_x86.cab', 'JUN2007_XACT_x64.cab', 'JUN2007_XACT_x86.cab', 'JUN2008_d3dx10_38_x64.cab', 'JUN2008_d3dx10_38_x86.cab', 'JUN2008_d3dx9_38_x64.cab', 'JUN2008_d3dx9_38_x86.cab', 'JUN2008_X3DAudio_x64.cab', 'JUN2008_X3DAudio_x86.cab', 'JUN2008_XACT_x64.cab', 'JUN2008_XACT_x86.cab', 'JUN2008_XAudio_x64.cab', 'JUN2008_XAudio_x86.cab', 'Jun2010_D3DCompiler_43_x64.cab', 'Jun2010_D3DCompiler_43_x86.cab', 'Jun2010_d3dcsx_43_x64.cab', 'Jun2010_d3dcsx_43_x86.cab', 'Jun2010_d3dx10_43_x64.cab', 'Jun2010_d3dx10_43_x86.cab', 'Jun2010_d3dx11_43_x64.cab', 'Jun2010_d3dx11_43_x86.cab', 'Jun2010_d3dx9_43_x64.cab', 'Jun2010_d3dx9_43_x86.cab', 'Jun2010_XACT_x64.cab', 'Jun2010_XACT_x86.cab', 'Jun2010_XAudio_x64.cab', 'Jun2010_XAudio_x86.cab', 'Mar2008_d3dx10_37_x64.cab', 'Mar2008_d3dx10_37_x86.cab', 'Mar2008_d3dx9_37_x64.cab', 'Mar2008_d3dx9_37_x86.cab', 'Mar2008_X3DAudio_x64.cab', 'Mar2008_X3DAudio_x86.cab', 'Mar2008_XACT_x64.cab', 'Mar2008_XACT_x86.cab', 'Mar2008_XAudio_x64.cab', 'Mar2008_XAudio_x86.cab', 'Mar2009_d3dx10_41_x64.cab', 'Mar2009_d3dx10_41_x86.cab', 'Mar2009_d3dx9_41_x64.cab', 'Mar2009_d3dx9_41_x86.cab', 'Mar2009_X3DAudio_x64.cab', 'Mar2009_X3DAudio_x86.cab', 'Mar2009_XACT_x64.cab', 'Mar2009_XACT_x86.cab', 'Mar2009_XAudio_x64.cab', 'Mar2009_XAudio_x86.cab', 'Nov2007_d3dx10_36_x64.cab', 'Nov2007_d3dx10_36_x86.cab', 'Nov2007_d3dx9_36_x64.cab', 'Nov2007_d3dx9_36_x86.cab', 'NOV2007_X3DAudio_x64.cab', 'NOV2007_X3DAudio_x86.cab', 'NOV2007_XACT_x64.cab', 'NOV2007_XACT_x86.cab', 'Nov2008_d3dx10_40_x64.cab', 'Nov2008_d3dx10_40_x86.cab', 'Nov2008_d3dx9_40_x64.cab', 'Nov2008_d3dx9_40_x86.cab', 'Nov2008_X3DAudio_x64.cab', 'Nov2008_X3DAudio_x86.cab', 'Nov2008_XACT_x64.cab', 'Nov2008_XACT_x86.cab', 'Nov2008_XAudio_x64.cab', 'Nov2008_XAudio_x86.cab', 'Oct2005_xinput_x64.cab', 'Oct2005_xinput_x86.cab', 'OCT2006_d3dx9_31_x64.cab', 'OCT2006_d3dx9_31_x86.cab', 'OCT2006_XACT_x64.cab', 'OCT2006_XACT_x86.cab']
Where are we?  C:\Documents and Settings
OK, walking through:  C:\Documents and Settings
Traceback (most recent call last):
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 25, in <module>
    walkThisDir("C:\\")
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 19, in walkThisDir
    walkThisDir(os.path.join(topleveldir, d))
  File "C:\Users\v\workspace\os_walk_DIC_example\sr\mywalk.py", line 8, in walkThisDir
    for thing in os.listdir(topleveldir):
WindowsError: [Error 5] Access is denied: 'C:\\Documents and Settings/*.*'




Hey, wait a minute! Why you crash, Python?

Well, it turns out there are some things you just can't do without admin access. So, what should we do?

I know! try to do it, and if we can't, move on! That's the Python way!


#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    dirs = []
    files = []
    for thing in os.listdir(topleveldir):
        if os.path.isfile(os.path.join(topleveldir, thing)):
            print "added file ", thing
            files.append(thing)
        elif os.path.isdir(os.path.join(topleveldir, thing)):
            print "added dir ", thing
            dirs.append(thing)
    print "Directories here: ", str(dirs)
    print "Files here: ", str(files)
    for d in dirs:
        print "Where are we? ", os.path.join(topleveldir, d)
        try:
            walkThisDir(os.path.join(topleveldir, d))
        except Exception, e:
            print "LAME! but let's keep going. Message for those who care: ", e
        

    
    
if __name__ == "__main__":
    walkThisDir("C:\\")



Ok, well that's cool. We can print out all that crap, but that's not very useful!
Python has things called generators in them. Let's try one of those to give us the path to the directory we're in, the directories in this directory, and the files in this directory, and rercurse through all of them!

We can do that with the yield statement.


#We might try, say...
import os

def walkThisDir(topleveldir):
    print "OK, walking through: ", topleveldir
    dirs = []
    files = []
    try:
        for thing in os.listdir(topleveldir):
            if os.path.isfile(os.path.join(topleveldir, thing)):
                files.append(thing)
            elif os.path.isdir(os.path.join(topleveldir, thing)):
                dirs.append(thing)
    except Exception, e:
        print "Meh, admin access needed or so. ", e
            
    yield topleveldir, dirs, files
    for d in dirs:
        new_top = os.path.join(topleveldir, d)
        if os.path.isdir(new_top):
            for x in walkThisDir(new_top):
                yield x
    
    
if __name__ == "__main__":
    for root, dirs, files in walkThisDir("C:\\"):
        print root
        print dirs
        print files





SAWEEEET! it (mostly) works!

Of course, this example is totally crappy. That's why we use the tried and true library routines, because most, if not all, have at least 1 gigabyte of trash talking associated with them, with necessary improvements being made to reduce the amount of trash talking expected for the future.

I refer you back to help(os) to check out walk. It does what we just did, except better.


And, here you have:

import os    
if __name__ == "__main__":
    for root, dirs, files in os.walk("C:\\"):
        print root
        print dirs
        print files




NOW AINT THAT SOMETHING? BAM, Libraries my friend! Libraries got what you need! (or if they don't, make your own, read every gigabyte of badmouthing you get for your release, fix the issues, and eventually, you, too, can be awesome like Guido Van Rossum.










And for your viewing pleasure, please see <PYTHON DIR>\lib\os.py


On windows, probably
C:\python27\lib\os.py
On linux, probably
/usr/lib/python2.7/lib/os.py

Maybe /usr/local/lib.

You should be able to find it, though, with walk...

for root, dirs, files in os.walk("/"):
    if "os.py" in files:
        print root

or 

for root, dirs, files in os.walk("C:\\"):
    if "os.py" in files:
        print root




Also, be sure to check out shutil.py
or,

import shutil
help(shutil)

Lots of magic there! Also introduces you to the fact that some libraries are named in such a way as to make no sense to anyone. They faithfully hide their awesomeness, so you really need to browse every single one of them to see what goodies they hold. It's worth it, I promise.

This post has been edited by Python_4_President: 23 October 2012 - 06:43 PM

Was This Post Helpful? 3
  • +
  • -

#4 tpimtts  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 6
  • Joined: 17-October 12

Re: Confused! Help! Librarys! Programming concepts?

Posted 30 October 2012 - 05:31 PM

First of all let me thank you for an amazing response, that was a fantastic "walk" through! (pun intended) :)

I really appreciated and loved your style expressing your thought processes explaining why you would search through libraries/modules for useful code. I especially liked the flourish at the end where the reason and use of libraries really hits home when we see our "walkThisDir" has already been implemented in os.path! Then we see our finished code is merely 6 lines long!! :D Haha! Brilliant.

Thank you so much again for taking the time and effort to explain this to a newbie. I really am loving python at the minute, I have written a script to 'grep' through excel spreadsheets with the re module and format the output to stdout. It was a real trial and error work and it is very messy -- I would be interested to hear your thoughts on project development plans. I basically started out with a user prompt screen, which was a function that returned the users input to another function which opened/searched & formated the results from a specific spreadsheet depending on the match object returned. I wrote it in a way where I got the code working and then encapsulated it into functions to divide it up, it was a very un-organised method, and I'm sure it could have been done better.

I would love to hear your thought-processes from the point of having a large overall goal then dividing into problems/finding a problem --> implementing the solution. You touched on this in your last post, and would be very interested to hear more -- thanks again!


tpimtts.

PS. I thought you explained everything perfectly! :)
Was This Post Helpful? 0
  • +
  • -

#5 Python_4_President  Icon User is offline

  • D.I.C Regular

Reputation: 53
  • View blog
  • Posts: 321
  • Joined: 13-August 11

Re: Confused! Help! Librarys! Programming concepts?

Posted 31 October 2012 - 04:49 PM

View Posttpimtts, on 30 October 2012 - 05:31 PM, said:

First of all let me thank you for an amazing response, that was a fantastic "walk" through! (pun intended) :)

I really appreciated and loved your style expressing your thought processes explaining why you would search through libraries/modules for useful code. I especially liked the flourish at the end where the reason and use of libraries really hits home when we see our "walkThisDir" has already been implemented in os.path! Then we see our finished code is merely 6 lines long!! :D Haha! Brilliant.

Thank you so much again for taking the time and effort to explain this to a newbie. I really am loving python at the minute, I have written a script to 'grep' through excel spreadsheets with the re module and format the output to stdout. It was a real trial and error work and it is very messy -- I would be interested to hear your thoughts on project development plans. I basically started out with a user prompt screen, which was a function that returned the users input to another function which opened/searched & formated the results from a specific spreadsheet depending on the match object returned. I wrote it in a way where I got the code working and then encapsulated it into functions to divide it up, it was a very un-organised method, and I'm sure it could have been done better.

I would love to hear your thought-processes from the point of having a large overall goal then dividing into problems/finding a problem --> implementing the solution. You touched on this in your last post, and would be very interested to hear more -- thanks again!


tpimtts.

PS. I thought you explained everything perfectly! :)



Hmm..

I gotta think about how to pull that one off. Currently I could classify my code-writing into these categories:

"Emergency band-aid" applications (I work at a company with __REALLY__ stupid employees, so stupid, in fact, they make me look smart. It's tragic, really. Often I prevent companies from losing many thousands of dollars by automagically fixing problems caused by careless programmers that either work for my company, their company, or some other company. It's soooo scary being me in these situations.)

"Automation-For-Better-Living" applications (Mundane tasks are no problem to automate in Python, generally, and I do lots of that)

"Can I do that" applications (Every now and then I get a whacky idea, like building an RC car, powered by Arduino/Xbee and controlled by a laptop running a Python script to catch keypresses and do something interesting with them, like steering, throttle, lights, and acceleration)

"Just for fun" applications (I play Armed Assault 2 and am working on some neat things for that, later I'll probably make a similar post containing a question not entirely unlike yours as to how to approach writing a python-to-sqf generator/compiler/whatever you might call it (sqf is a scripting language in Arma, and it really sucks))

and "I need this even though I hate it" applications (like ftp servers and such)



Ultimately, I would want to make the attempt in such a way as to goad Baavgai (resident super-ninja, probably capable of writing an OS with one hand, while blindfolded, in an afternoon, without ever compiling anything until the very end) into giving a way better, way more valuable response. I have done that twice now.


The best one yet is: converting from Python to Java

Another good one is: Baavgai compulsively rewrites my code I wrote to help someone else


However, I don't always succeed..

I failed here, somehow (I suspect it was because of the RTFM-ness of the question). I ended up being the only one responding to the thread: Python_4_President should read more manuals


I thoroughly failed here, probably due to verbosity/lack of clarity/obscure nature of the topic/flouride in the water: 0 Replies for EDI



Generally, though, I find myself building and testing ideas related to what I want to accomplish, then seeing the bigger picture, then clumsying my way through integrating all the components, and THEN I see (most of) what I want to do and redo everything from scratch, more or less. It's a slow, painful, but often fruitful approach that gets me through for now.

I think, probably, once I've gone through all of that for another year or so, I will have designed and implemented enough things to be well versed in things I've done that actually work well, and have a toolbox of patterns and approaches that don't suck so much, which will probably speed up my development times.

I'd REALLY love to see Baavgai dump his noggin, though. I was pretty impressed with myself after he said I gave him an idea. I suspect that doesn't happen very often.

I'm going to sleep on this for a bit, and see what dreams may come.

This post has been edited by Python_4_President: 31 October 2012 - 05:11 PM

Was This Post Helpful? 0
  • +
  • -

#6 Python_4_President  Icon User is offline

  • D.I.C Regular

Reputation: 53
  • View blog
  • Posts: 321
  • Joined: 13-August 11

Re: Confused! Help! Librarys! Programming concepts?

Posted 06 November 2012 - 01:55 PM

Ok, I think I figured out a cool way to handle this. Since I have recently developed a great example, we'll use that.

To start, a self classification: I am somewhere between novice and professional. I've written code (mostly Python!) for fun and profit, for most hours of every day I've been alive, for the last couple of years. I try to follow the motto, "First make it work, then make it right, then make it fast." Unfortunately, most of my projects never make it past the "Make it work" phase, and a lot don't even get there completely.

What I want to do with this is:
A.) Clarify my intent and motivation for investing my time in this project.
B.) Elucidate my process for iteratively developing components of a non-trivial program.
C.) Deconstruct the monstrosity that results from building things before I know what I'm building.
D.) Develop and assemble quality components worthy of my effort, and your time studying it.

First: The situation
I play a game called Armed Assault 2 as part of a clan named Cerberus 5. Cerberus 5 hosts a PVP (player versus player) server with custom missions that, mostly, provide a terrific challenge to complete. It is one of the only servers available these days where a player can find such competitive and enjoyable gameplay on a daily basis.

Second: The problem
We've got a lot of great custom missions that are built on islands that are considered "Addons" because they do not come with the game and you won't find them in any patches, either. In order for a player to play these missions, they MUST have installed the addons and launched their game with parameters to enable them. If this has not been done, the user gets booted from the server by the game engine.

Third: The idea
If we had a tool compatible with Windows XP and Windows 7 (32bit and 64bit) and the standalone and STEAM versions of the game that would download and install required addons (and some other optional utilities that further increase the enjoyability of the player's experience), we would have better turnout when playing the missions we've made for the addon islands, and avoid situations where nearly 3/4 of the server gets booted when the map changes.

Fourth: The manual installation process (assuming the game is installed and patched already)
A.) Locate the ArmA2 installation directory
B.) Download Island Panthera (7z file, 185mb compressed)
C.) Download Island ToraBora (7z file, 215mb compressed)
D.) Download Island Nogova2010 (7z file, 218mb compressed)
E.) Extract Island Panthera to ArmA2 installation directory (requires administrator privileges)
F.) Extract Island ToraBora to ArmA2 installation directory (requires administrator privileges)
G.) Extract Island Nogova2010 to ArmA2 installation directory (requires administrator privileges)
H.) Create a shortcut to arma2oa.exe with these parameters: -mod=@panthera; -mod=@ToraBora; -mod=Nogova2010
I.) Launch ArmA2 via the shortcut
J.) Find the C5 server in the in-game server menu and join it
--OPTIONAL--
K.) Download TeamSpeak 3 (executable installer file, 30mb)
L.) Launch the TeamSpeak 3 installer (requires administrator privileges)
M.) Connect to the C5 TeamSpeak 3 server and specify a nickname

Fifth: Developer Challenges
A.) Locating the ArmA2 installation directory (In theory, there could be more than one, and they may not be on the same drive)
B.) Knowing what addons are required to play on the server uninterrupted (There could be more, or less in the future)
C.) Downloading the addons (links break, hosting providers change, etc)
D.) Extracting the addons (Most are 7z, some are zip, some are rar, etc)
E.) Creating the shortcut (Some users stick it all on Desktop. Some want it in the quicklaunch bar, some in a specific folder, etc)
F.) Keep the user informed of what changes are being made, and the statuses of each operation at all times.
G.) At the end of the process, inform the user of what was done, what he can do now, and thank him for using the tool and getting set up to play on the C5 server. Also provide general info about the organization, etc.
H.) Cleaning up any messes that were made.

Sixth: Judgement calls
A.) Do you create a generic ArmA2 shortcut that simply launches the game, or add more parameters to launch the game and connect to the C5 server automatically, and with a specific profile?
B.) Do you create a shortcut for TeamSpeak 3 which allows the user to launch the executable and connect to the C5 TeamSpeak 3 server with a specified user name?
C.) Where do you put the initial downloaded files? Eventually we'll need to delete them.
D.) The downloads are large, Should you allow the user to download as many addons at a time as he prefers?

Seventh: Points of failure
A.) Fail to locate ArmA2 installation directory
B.) Fail to download addon
C.) Fail to extract addon
D.) Fail to create shortcut
E.) Fail to install additional utility
F.) Fail to inform user

Eight: Personal uncertainties
A.) The example I have is around 1000 lines of pretty confusing code. How should I expose it?
B.) Should this stay in this thread, or go in a new one?

Ninth: The motivation
Seeing as how the C5 server is the only one I __EVER__ play on, it would be nice if it __ALWAYS__ had a lot of people in it. Currently, the right time to be on is around 8:45PM CST. It stays real good until about 11:15PM CST. I believe, as a skilled developer, that I can build the things necessary to create a crack-like addictive property of the server to keep the players fiending for more at all hours of the day and night (technically still day for the russkies that come on, but you get the idea). Initially, however, I just want to get this installer working. The missions on the addon islands totally rock and should get more play time (they're benched at the moment, because most players don't know how to install them and don't have the patience to learn.)



Considering the above, are you down for this idea? I think it would help me as much as it might help you (and any one else that reads the thread). I'd get more experience at trying to do things the right way (IE, the "make it right" part. The example already works..ish...) You'd get to see some terrible code like you might right one day get sent back through the brain and come out a least much better, possibly a whole lot better, and how it got that way.

All bets are off on what the old timers might get out of it. Maybe a headache.

This post has been edited by Python_4_President: 06 November 2012 - 02:14 PM

Was This Post Helpful? 0
  • +
  • -

Page 1 of 1