I would like to iterate over a stream of words, say, from STDIN or a file (or any random input stream). Typically, this is done like this,
def iter_words(f): for line in f: for word in line.split(): yield word
And then one can simply,
for word in iter_words(sys.stdin): # do something
For a more concrete example, let's say we need to keep a count of every unique word in an input stream, something like this,
from collections import Counter c = Counter for word in iter_words(sys.stdin): c.update([word])
The only problem with this approach is that it will read data in line-by-line, which in most cases is exactly what we want, however, in some cases we don't have line-breaks. For extremely large data streams we will simply run out of memory if we use the above generator.
Instead, we can use the read() method to read in one-byte at a time, and manually construct the words as we go, like this,
def iter_words(sfile): chlist =  for ch in iter(lambda: sfile.read(1), ''): if str.isspace(ch): if len(chlist) > 0: yield ''.join(chlist) chlist =  else: chlist.append(ch)
This approach is memory efficient, but extremely slow. If you absolutely need to get the speed while still being memory efficient, you'll have to do a buffered read, which is kind of an ugly hybrid of these two approaches.
def iter_words(sfile, buffer=1024): lastchunk = '' for chunk in iter(lambda: sfile.read(buffer), ''): words = chunk.split() lastchunk = words[-1] for word in words[:-1]: yield word newchunk =  for ch in sfile.read(1): if str.isspace(ch): yield lastchunk + ''.join(newchunk) break else: newchunk.append(ch)