0 Replies - 2107 Views - Last Post: 22 April 2015 - 08:05 PM

#1 atraub   User is offline

  • Pythoneer
  • member icon

Reputation: 830
  • View blog
  • Posts: 2,251
  • Joined: 23-December 08

Lazily Read Files With a Generator (Great for BIG files)

Posted 22 April 2015 - 08:05 PM

This function will open a file given a path and will read that file in "Chunks". The default size is 1024 but if it's reading slowly you may want to consider upping that number based on what your environment can handle. Because of the style being used, there's no need to worry about closing the file manually. It'll close when the generator is finished reading it.
def file_read_generator(filePath, chunk_size=1024):
    with open(filePath) as myFile:
        while True:
            data = myFile.read(chunk_size)
            if not data:
                break
            yield data  



Example Usage
 
myFile = file_read_generator("./test.txt")
with open("./test.txt", "r") as outFile:
    for chunk in myFile:
       outFile.write(chunk.replace("the","peanut butter"))


one problem you could run into is this: What if the word you what to replace is cut off by the chunk? Issues like this are easy enough to get around with fancy footwork, but it is something to bear in mind.

This post has been edited by atraub: 23 April 2015 - 06:59 AM


Is This A Good Question/Topic? 1
  • +

Page 1 of 1