Hi,
I am comparing 2000 files with one other file. I want the program to go through each line in both files and compare. If the line is present, then it has to write to another file. What I tried was to open both the files and use readlines() to read into an list. Then I used for loop like this:

The above code is only a chunk btw. My problem is that both the files are not of the same size so I feel using zip or izip is not ideal in this situation. Is there any other solution where in I can compare line by line between both the files and iterate the complete file.

Thanks in advance,
Cheers,
Chav.

P.S: Though this is an assignment, I am not asking for the code (unless it is too complex). Please let me know any ideas or modules that can help me. Also both the files are huge with around 2000 lines each.

This post has been edited by chavanak: 3 Nov, 2009 - 06:27 AM

There is a version of izip, called izip_longest that may suit: http://docs.python.org/library/itertools.h...ls.izip_longest

Here's my refactoring of your code, if it helps.

def getFileLines(fileName):
	fh = open (fileName, "r") 
	lines = fh.readlines()
	fh.close()
	return map(string.strip, lines)

splitter = [ s.split('\t') for s in getFileLines("1complex.txt") ]
chain_sep = [ os.path.basename(file) for file in os.listdir('.') if file.endswith('.pd') ]

for (i,s) in izip(chain_sep,splitter):
	fopen_7=open (i+"r.pdb", "w")
	fopen_8=open (i+"l.pdb", "w")
	for (a,y) in izip_longest(getFileLines(i), getFileLines(s[0]+".cr"), None):
		if a[0:4]=="ATOM":
			if a[21] == "R":
				print >>fopen_7, a
			elif y!=None and a[7:13]==y[7:13]:
				print >>fopen_8, a
	fopen_7.close()
	fopen_8.close()

If not, well, it really shouldn't be too hard to roll your own.

Hi,
Thanks for the reply. Though izip_longest provides the necessary solution for me, I cannot use it on my work computer since it is pegged at python version 2.4 and izip_longest is a 2.6 feature

So is it possible to do it any other way. Please do let me know
Cheers,
Chav

"My library doesn't have that function so I give up" is not a programmer's attitude. Rather, the mindset should be "in the absence of a library, I get to write my own function to suit my needs."

In this instance, the function is almost trivial and shouldn't take more than a few minutes to put together. About five, actually.

The izip and izip_longest functions are iterators. Here's my take on what I'd want from izip_longest:

def myZip(list1, list2):
	if len(list1)<len(list2):
		smallest = len(list1)
	else:
		smallest = len(list2)
	for i in range(smallest): yield (list1[i], list2[i])
	for v in list1[smallest:]: yield (v, None)
	for v in list2[smallest:]: yield (None, v)

a1 = ['x'+str(i) for i in range(5) ]
a2 = ['y'+str(i) for i in range(8) ]
print [i for i in myZip(a1, a2)]

Result:

Hi,
Thanks for the help. I do understand that if there is no library, you gotto create one but this is mostly the last time I will be using python and also I have just started with it I did try out your solution by making changes as I wanted but I failed at getting what I want Just to expand a bit on my problem.



As you see in the above piece of data, the only part that is common to both files is the one in bold (the above is just a chunk of a code). So ideally I am supposed to compare the bold data from file 1 and if it exists in file 2, I have to retain it and remove the remaining data.
For e.g.:


If the above entry is there in both files then I gotto retain it in file-2 and remove all other entries. I tried to add the required list position to the sample code you gave me but I failed to get the results. Please let me know if I can differentiate the above data and if so how can I do it? I tried the same in perl and I am able to do it very easily but the same in python is becoming tougher for me as I am very new to python (learning for the past week or so)
Cheers,
Chav

I'm not really following your logic. However, I can offer this:

Don't think about processing files. File processing is a pain. Rather, in python in particular, think about processing lists.

The following code takes a pair of lists and marks them based on matching criteria and if the value is valid. From this you may be able to find something applicable to your project. Remember, don't be afraid of functions; they add organization and to some extent are self documenting. I use a number of "list comprehensions" here. If you don't know what they are, learn them well; they are a common python technique.

listDataTypes = ['Unknown', 'Junk', 'Unique', 'Both']

def processLists(list1, list2):
	def markAll(plist, markValue):
		count = 0
		for item in plist:
			item[1] = markValue
			count += 1
		return count
			
	def markJunk(plist):
		return markAll([v for v in plist if not v[0][0:4]=="ATOM" ], 1)

	def markBoth(pList1, pList2):
		def markMatch(plist, value, markValue):
			return markAll([v for v in plist if v[0]==value ], markValue)
				
		for item in pList1:
			if item[1]==0:
				value = item[0]
				if not markMatch(pList2, value, 3)==0:
					markMatch(pList1, value, 3)
				else:
					markMatch(pList1, value, 2)

	# make a list of the values with flags, set to unknown
	pList1 = [ [s, 0] for s in list1 ]
	pList2 = [ [s, 0] for s in list2 ]
	
	# exclude junk
	markJunk(pList1)
	markJunk(pList2)
	
	# find matches
	markBoth(pList1, pList2)
	
	# those left in pList2 are unique
	markAll([v for v in pList2 if v[1]==0], 2)
	
	return (pList1, pList2)



def test():
	def showList1(list, name):
		print "---", name
		for item in list:
			print "\t", item
		print "---\n"

	def showList2(list, name):
		print "---", name
		for item in list:
			print "%s\t%s" % (listDataTypes[item[1]], item[0])
		print "---\n"

	d1 = ['ATOM   123','junk data','ATOM   213','ATOM   132']
	d2 = ['ATOM   232','ATOM   123','junk data','ATOM   321']
	showList1(d1, 'list (or file contents) 1')
	showList1(d2, 'list (or file contents) 2')
	
	(pd1, pd2) = processLists(d1, d2)
	showList2(pd1, 'processed list 1')
	showList2(pd2, 'processed list 2')

test()

Results:


Hope this helps.