[wisam abbasi]

Hi
I have a problem when trying to read files that have Arabic language characters stored in using Python
Can anyone help please?

[Simown]

The files are stored with Unicode encoding?

Then I suggest:
Reading and Writing Unicode Data

[Motoma]

Could you post your code? You may need to specify that you want to read your file in binary mode: myfile = open('filename.txt', 'rb')

[wisam abbasi]

Simown, on 24 January 2011 - 09:49 AM, said:

The files are stored with Unicode encoding?

Then I suggest:
Reading and Writing Unicode Data

Yes they are, I've tried to store them with ANSI encoding and it worked. The problem now is how to change the encoding using Python not the editor????
Thanks for help

Motoma, on 24 January 2011 - 09:50 AM, said:

Could you post your code? You may need to specify that you want to read your file in binary mode: myfile = open('filename.txt', 'rb')

input = open('wisam3.txt', 'r')
text=input.read()

Thanks

[Nallo]

When you read a text file you need to tell Python which encoding was used:

wisamfile = open('wisam3.txt', 'r')
text = wisamfile.read()
wisamfile.close()

#Text is a bytestring. If there are non-ascii characters in
#Python cannot know what they mean.
#So you need to tell python which codec to use.
#Lets turn it into a unicode string,
#asuming the file was encoded in utf-8:
text_unicode = unicode(text, 'utf-8')

#now do something with your unicode text
print text_unicode

#lets save it with a different encoding (iso8859_6)
#Just for showing. I suggest you stick to utf-8 when working with unicode
text_iso8859_6 = text_unicode.encode('iso8859_6')
wisam_arab = open('wisam3_arabencoding.txt', 'w')
wisam_arab.write(text_iso8859_6)
wisam_arab.close()

[wisam abbasi]

Nallo, on 07 February 2011 - 05:17 AM, said:

When you read a text file you need to tell Python which encoding was used:

wisamfile = open('wisam3.txt', 'r')
text = wisamfile.read()
wisamfile.close()

#Text is a bytestring. If there are non-ascii characters in
#Python cannot know what they mean.
#So you need to tell python which codec to use.
#Lets turn it into a unicode string,
#asuming the file was encoded in utf-8:
text_unicode = unicode(text, 'utf-8')

#now do something with your unicode text
print text_unicode

#lets save it with a different encoding (iso8859_6)
#Just for showing. I suggest you stick to utf-8 when working with unicode
text_iso8859_6 = text_unicode.encode('iso8859_6')
wisam_arab = open('wisam3_arabencoding.txt', 'w')
wisam_arab.write(text_iso8859_6)
wisam_arab.close()

Thanks alot, you were really helpfull but, can I know the text file encoding automatically using python without going back to open the file because I need that in my project?

Thanks again

[Nallo]

Unfortunately there is no way to be sure about the used encoding of a text file. This is not python specific it applies to any programming languagge. A text file after all is just a secuence of bytes. Only when you know the encoding used those bytes have meaning.

You could use some heuristics to check the text files and use the most likely encoding candidate. But it is unreliable and messy (and I am not an expert with this, so don't expect any advice from me)

As for arabic letters. Chances are decent that either iso8859_6 (a 1Byte per character encoding extending ascii) or utf-8 (the most used encoding for unicode) were used (again I am not an expert with that)

[wisam abbasi]

Nallo, on 07 February 2011 - 08:03 AM, said:

Unfortunately there is no way to be sure about the used encoding of a text file. This is not python specific it applies to any programming languagge. A text file after all is just a secuence of bytes. Only when you know the encoding used those bytes have meaning.

You could use some heuristics to check the text files and use the most likely encoding candidate. But it is unreliable and messy (and I am not an expert with this, so don't expect any advice from me)

As for arabic letters. Chances are decent that either iso8859_6 (a 1Byte per character encoding extending ascii) or utf-8 (the most used encoding for unicode) were used (again I am not an expert with that)

Ok, but the code you've wrote raises an error when trying to create an Arabic text file
it prints the text on the screan but does not create the file
here is the error:

Traceback (most recent call last):
  File "C:\Users\one.omary\Desktop\Asummerizer.py", line 36, in <module>
    text_iso8859_6 = text_unicode.encode('iso8859_6')
  File "C:\Python25\lib\encodings\iso8859_6.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character u'\ufeff' in position 0: character maps to <undefined>

This post has been edited by atraub: 07 February 2011 - 09:13 AM
Reason for edit:: code tags for readability

[Nallo]

Sorry, I am at a loss here

The feff is the byte order mark for encoded unicode files, but I dont understand why it shows up here.

Can you show me the whole code you used to produce that error? Obviously you merged my sniplet into your code but without seeing it I won't understand what went wrong.

And another question: Are you using Python 2.x or 3.x? I made my posts assuming Python 2.x. But dealing with strings is one of the few things that considerably changed in Python 3.x.

[wisam abbasi]

Nallo, on 07 February 2011 - 09:13 AM, said:

Sorry, I am at a loss here

The feff is the byte order mark for encoded unicode files, but I dont understand why it shows up here.

Can you show me the whole code you used to produce that error? Obviously you merged my sniplet into your code but without seeing it I won't understand what went wrong.

And another question: Are you using Python 2.x or 3.x? I made my posts assuming Python 2.x. But dealing with strings is one of the few things that considerably changed in Python 3.x.

Ok, here is the code:

from __future__ import division
import re
import math
from array import array


input = open('wisam4.txt', 'r')
text=input.read()
input.close()


text_unicode = unicode(text, 'utf-8')
print text_unicode
text_iso8859_6 = text_unicode.encode('iso8859_6')  

wisam_arab = open('wisam3_arabencoding.txt', 'w')  

wisam_arab.write(text_iso8859_6)  

wisam_arab.close() 

and here is the error:

Traceback (most recent call last):
  File "C:\Users\one.omary\Desktop\2.py", line 14, in <module>
    text_iso8859_6 = text_unicode.encode('iso8859_6')
  File "C:\Python25\lib\encodings\iso8859_6.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character u'\ufeff' in position 0: character maps to <undefined>

And I am using python 2.x

[Nallo]

I googled around a bit and found the unicode FAQ.

It seems the first bytes of an encoded unicode text file may or may not contain the byte order marker. If it is there it indicates which encoding was used. It is not part of the text, just tells about the encoding.

If it is there you can strip it from the string to avoid the encoding error produced.

Sorry I am not all that helpful here.

[wisam abbasi]

Nallo, on 07 February 2011 - 09:44 AM, said:

I googled around a bit and found the unicode FAQ.

It seems the first bytes of an encoded unicode text file may or may not contain the byte order marker. If it is there it indicates which encoding was used. It is not part of the text, just tells about the encoding.

If it is there you can strip it from the string to avoid the encoding error produced.

Sorry I am not all that helpful here.

Thanks Nallo, you were really helpful