12 Replies - 537 Views - Last Post: 21 June 2012 - 02:56 PM Rate Topic: -----

#1 kingAmin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 6
  • Joined: 20-June 12

replacing missing dat with zeros

Posted 20 June 2012 - 08:31 AM

Hi! I'm working on two data files one has missing data, I'm interested in writing a python code to read the complete file put zeros where the other file has missing values.Can any one help me with a code? Thank you


Is This A Good Question/Topic? 0
  • +

Replies To: replacing missing dat with zeros

#2 atraub  Icon User is offline

  • Pythoneer
  • member icon

Reputation: 759
  • View blog
  • Posts: 2,010
  • Joined: 23-December 08

Re: replacing missing dat with zeros

Posted 20 June 2012 - 09:55 AM

Sure, we can help you, show us what you have so far.
Was This Post Helpful? 0
  • +
  • -

#3 kingAmin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 6
  • Joined: 20-June 12

Re: replacing missing dat with zeros

Posted 20 June 2012 - 01:30 PM

Hi! I'm working on two data files one has missing data, I'm interested in writing a python code to read the complete file put zeros where the other file has missing values.Can any one help me with a code? Thank you
from numpy import*
fp = open("beth.txt","r")
fs = open("bftn.txt","r")
f = open("newdata.out","w")
a = []
b = []
c = []
n = 0
for line in fp:
    a.append( int(line.split()[0]))
    b.append( int(line.split()[1]))
    c.append( float(line.split()[2]))
    n += 1
fp.close()

eps = 1.0e-5
for line in fs:
  a1 = int(line.split()[0])
  b1 = int(line.split()[1])
  c1 = float(line.split()[2])
#  Test for match of all three numbers
  ok = 0
  for i in range(n):
    if a1 == a[i] and b1 == b[i]:
      if abs(c1 -c[i]) < eps:
        ok = 1
# If there is a match ok = 1, otherwise ok = 0
  if ok == 1:
    f.write("%d\t%d\t%f\n" %(a[i],b[i],c[i]))
  else:
    f.write("%d\t%d\t%s\n" %(a1,b1,"nan"))

fs.close()
f.close()


Edited by Dogstopper: :code:

This post has been edited by atraub: 20 June 2012 - 07:50 PM

Was This Post Helpful? 0
  • +
  • -

#4 atraub  Icon User is offline

  • Pythoneer
  • member icon

Reputation: 759
  • View blog
  • Posts: 2,010
  • Joined: 23-December 08

Re: replacing missing dat with zeros

Posted 20 June 2012 - 07:50 PM

Posts merged. It's not necessary, or appropriate, to create a duplicate topic just to add more information.

Could you give us an example of the inputs and an example of the output?

This post has been edited by atraub: 21 June 2012 - 06:38 PM

Was This Post Helpful? 0
  • +
  • -

#5 kingAmin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 6
  • Joined: 20-June 12

Re: replacing missing dat with zeros

Posted 21 June 2012 - 01:00 AM

Hi! I'm working on two data files one has missing data, I'm interested in writing a python code to read the complete file put zeros where the other file has missing values.Can any one help me with a code? This the code i have written
from numpy import*


fp = open("beth.txt","r")


fs = open("bftn.txt","r")


f = open("newdata.out","w")

a = []

b = []

c = []

n = 0

for line in fp:
    
    a.append( int(line.split()[0]))
    
    b.append( int(line.split()[1]))
    
    c.append( float(line.split()[2]))
    
    n += 1
fp.close()
for i in range(n):

 
    for line in fs:
    
        a1 = int(line.split()[0])
    
        b1 = int(line.split()[1])

    
    
        if a1 == a[i] or b1 == b[i]:

       
           f.write("%d\t%d\t%f\n" %(a[i],b[i],c[i]))

    
        else:
       
           f.write("%d\t%d\t%s\n" %(a1,b1,"nan"))



fs.close()

f.close()


and this is what i'm getting
Attached File  result.txt (606K)
Number of downloads: 30
Was This Post Helpful? 0
  • +
  • -

#6 atraub  Icon User is offline

  • Pythoneer
  • member icon

Reputation: 759
  • View blog
  • Posts: 2,010
  • Joined: 23-December 08

Re: replacing missing dat with zeros

Posted 21 June 2012 - 07:57 AM

Stop copying and pasting the same damn introduction. I was very clear:

1.) Show us what you would input to the program
2.) Show us what the program would then output.

If this request is beyond your abilities, then helping you is beyond mine.

This post has been edited by atraub: 21 June 2012 - 07:58 AM

Was This Post Helpful? 0
  • +
  • -

#7 kingAmin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 6
  • Joined: 20-June 12

Re: replacing missing dat with zeros

Posted 21 June 2012 - 09:34 AM

FOR EXAMPLE THESE ARE THE TWO INPUT DATA FILES AND ONE OUTPUT DATA FILE
   
INPUT DATA FILES                                                OUTPUT DATA FILE

     A                                   B                         IWANT MAAKE "B" START FROM FIRST DAY AS "A"
 DAY NO. TIME IN HRS.  TEC/U     DAY NO. TIME IN HRS..  TEC/U      DAY NO. TIME IN HRS.  TEC/U
 1       1.0           4.5       2        1.00          5.3        1           1.0        NAN
 1       2.0           3.5       2        2.00          4.5        1           2.0        NAN
 1       23.59         5.6       2        3.0           4.2        1           23.59      NAN
 2       1.0           3.5       3        1.0           2.5        2           1.0        5.3
 2       2.0           2.5       3        2.0           3.9        2           2.0        4.5
 2       3.0           3.2       3        23.59         5.3        2           3.0        4.2
 3       1.0           1.5       4        1.0           9.4        3           1.0        2.5
 3       2.0           5.3       .          .             .        3           2.0        3.9
 3       23.59         7.2       .          .             .        3           23.59      5.3
 4       1.0           3.7       .          .             .        4           1.0        9.4
 4       2.0           6.2       29       23.59          3.5       4           2.0        NAN
 4       23.59         4.3                                         4           23.59      NAN
 .       .              .                                          .             .         .
 .       .              .                                          .             .         .
 .       .              .                                          .             .         .
 29       23.59        4.2                                         29           23.59     3.5
 30      1.0           6.2                                         30           1.0       NAN

This post has been edited by atraub: 21 June 2012 - 10:00 AM
Reason for edit:: added code tags to try to improve readability

Was This Post Helpful? 0
  • +
  • -

#8 atraub  Icon User is offline

  • Pythoneer
  • member icon

Reputation: 759
  • View blog
  • Posts: 2,010
  • Joined: 23-December 08

Re: replacing missing dat with zeros

Posted 21 June 2012 - 09:59 AM

looking at your example, I can't figure out what you're looking for.
Was This Post Helpful? 0
  • +
  • -

#9 kingAmin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 6
  • Joined: 20-June 12

Re: replacing missing dat with zeros

Posted 21 June 2012 - 09:59 AM

FOR EXAMPLE THESE ARE THE TWO INPUT DATA FILES AND ONE OUTPUT DATA FILE

[/ INPUT DATA FILE

A
DAY NO. TIME IN HRS. TEC/U
1 1.0 4.5
1 2.0 3.5
1 23.59 5.6
2 1.0 3.5
2 2.0 2.5
2 3.0 3.2
3 1.0 1.5
3 2.0 5.3
3 23.59 7.2
4 1.0 3.7
4 2.0 6.2
4 23.59 4.3
. . .
. . .
. . .
29 23.59 4.2
30 1.0 6.2]





[/ INPUT DATA FILE
B
DAY NO. TIME IN HRS.. TEC/U
2 1.00 5.3
2 2.00 4.5
2 3.0 4.2
3 1.0 2.5
3 2.0 3.9
3 23.59 5.3
4 1.0 9.4
. . .
. .
. . .
29 23.59 3.5]

[/ OUTPUT DATA FILE

DAY NO. TIME IN HRS. TEC/U
1 1.0 NAN
1 2.0 NAN
1 23.59 NAN
2 1.0 5.3
2 2.0 4.5
2 3.0 4.2
3 1.0 2.5
3 2.0 3.9
3 23.59 5.3
4 1.0 9.4
4 2.0 NAN
4 23.59 NAN
. . .
. . .
. . .
29 23.59 3.5
30 1.0 NAN]
Was This Post Helpful? 0
  • +
  • -

#10 baavgai  Icon User is offline

  • Dreaming Coder
  • member icon

Reputation: 5829
  • View blog
  • Posts: 12,683
  • Joined: 16-October 07

Re: replacing missing dat with zeros

Posted 21 June 2012 - 11:10 AM

You do not want to screw with files. Read both entire files into memory. Use readlines, it will get you a list. Write a parse function for the line. You have three pieces of data. However, the first two form a key. You can use that key to compare both data sets and put them in order. A dictionary would help here.

Write a parse function that returns either valid line data or None.
Was This Post Helpful? 0
  • +
  • -

#11 kingAmin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 6
  • Joined: 20-June 12

Re: replacing missing dat with zeros

Posted 21 June 2012 - 11:33 AM

I DID SOMETHING LIKE THIS
fp = open("beth.txt","r")


fs = open("bftn.txt","r")


f = open("newdata.out","w")

a = []

b = []

c = []

n = 0

for line in fp:

a.append( int(line.split()[0]))

b.append( int(line.split()[1]))

c.append( float(line.split()[2]))

n += 1
fp.close()
for i in range(n):


for line in fs:

a1 = int(line.split()[0])

b1 = int(line.split()[1])



if a1 == a[i] or b1 == b[i]:


f.write("%d\t%d\t%f\n" %(a[i],b[i],c[i]))


else:

f.write("%d\t%d\t%s\n" %(a1,b1,"nan"))



fs.close()

f.close()


BELOW IS A PORTION OF THE RESULT I'M GETTING. WHICH IS PICKING ONLY ONE VALUE FROM THE THIRD COLUMN

233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
233 4 5.280000
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan
234 0 nan


This post has been edited by atraub: 21 June 2012 - 12:16 PM

Was This Post Helpful? 0
  • +
  • -

#12 atraub  Icon User is offline

  • Pythoneer
  • member icon

Reputation: 759
  • View blog
  • Posts: 2,010
  • Joined: 23-December 08

Re: replacing missing dat with zeros

Posted 21 June 2012 - 12:16 PM

1.) :code:
2.) Stop writing in caps
3.) Was that a question?

This post has been edited by atraub: 21 June 2012 - 12:17 PM

Was This Post Helpful? 0
  • +
  • -

#13 baavgai  Icon User is offline

  • Dreaming Coder
  • member icon

Reputation: 5829
  • View blog
  • Posts: 12,683
  • Joined: 16-October 07

Re: replacing missing dat with zeros

Posted 21 June 2012 - 02:56 PM

So... a, b, and c are parallel arrays? And you're casing values like 23.59 into int? Doing a split three times, never doing a validation check? The n variable is meaningless. Without indents, the code is crap. Your methodology is unfortunate.

Your second pass considers the second file, but ignores the first.
# pointless
# for i in range(n):
# might as well just say
i = 0
for line in fs:
	# when this loop is finished, you're out of file



Read them both first, then mess with them. Use functions. The same code should be used to read both files. You needn't do any casting to fit the files togehter.

Good luck.
Was This Post Helpful? 1
  • +
  • -

Page 1 of 1