9 Replies - 453 Views - Last Post: 07 May 2020 - 11:41 AM Rate Topic: -----

#1 fisicx   User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 57
  • Joined: 24-February 12

Special Characters not recognised

Posted 07 May 2020 - 12:49 AM

I have a CSV I want to import into an array and then save to a Wrordpress option.

The CSV is saved using OpenOffice as a Unicode (UTF-8) file.

I'm using fgetcsv to get the file contents

But if any of the cells have accents or diacritical marks (for example or ) the import fails.

A var_dump of the array returns the replacement character symbol in place of the required character.

When I remove the characters the import works.

What am I missing?

Is This A Good Question/Topic? 0
  • +

Replies To: Special Characters not recognised

#2 Ornstein   User is offline

  • D.I.C Head

Reputation: 105
  • View blog
  • Posts: 216
  • Joined: 13-May 15

Re: Special Characters not recognised

Posted 07 May 2020 - 02:23 AM

When you say "the import fails" what exactly do you mean? Presumably fgetcsv is successfully reading the row if you're able to var_dump the array - so do you mean it's "failing" because you're seeing errors in the output?

When you output the data, are you specifying that the output is UTF-8?

You may also want to look up the documentation for fgetcsv and pay attention to the note RE: the locale. You might for example experience some issues if your locale is set to something other than UTF-8 (e.g. ISO-8859-1).
Was This Post Helpful? 1
  • +
  • -

#3 fisicx   User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 57
  • Joined: 24-February 12

Re: Special Characters not recognised

Posted 07 May 2020 - 02:55 AM

Sorry, didn't explain myself well.

The import from the CSV works (because I can see the array) but it doesn't save the data to the database.

Remove the accents and diacritical and it saves to the database.

Locale isn't an issue.
Was This Post Helpful? 0
  • +
  • -

#4 Ornstein   User is offline

  • D.I.C Head

Reputation: 105
  • View blog
  • Posts: 216
  • Joined: 13-May 15

Re: Special Characters not recognised

Posted 07 May 2020 - 03:01 AM

Then you perhaps want to look at the encoding being used when communicating with the database and for storing the data.

If you want further guidance, you might want to post more info about your database, your code for connecting and querying, etc.
Was This Post Helpful? 0
  • +
  • -

#5 fisicx   User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 57
  • Joined: 24-February 12

Re: Special Characters not recognised

Posted 07 May 2020 - 03:50 AM

It's a standard wordpress sql database.

Within the database there is a table called wp_options which is where all options are stored.

There are normally no restrictions as to what you include. Accents and diacritical are permitted. For example: 'my name is <b>ń</b>' will be written exactly like that.

In the CSV there is: Crdit Agricole Festhypothek in unicode (utf-8)

When imported this becomes:

[0] => Array
        (
            [0] => Cr�dit Agricole Festhypothek 
        )

Was This Post Helpful? 0
  • +
  • -

#6 Ornstein   User is offline

  • D.I.C Head

Reputation: 105
  • View blog
  • Posts: 216
  • Joined: 13-May 15

Re: Special Characters not recognised

Posted 07 May 2020 - 04:54 AM

Are you using WP functions to access the database?

Do you pass your data through any other functions (e.g. htmlspecialchars)?

There's a few stages of the process where character encoding can cause issues. I wrote up a quick list in another thread - if you want to check over that.
Was This Post Helpful? 1
  • +
  • -

#7 CTphpnwb   User is offline

  • D.I.C Lover
  • member icon

Reputation: 3837
  • View blog
  • Posts: 13,994
  • Joined: 08-August 08

Re: Special Characters not recognised

Posted 07 May 2020 - 06:11 AM

What is the character set for the table?

https://stackoverflo...table-column-is
Was This Post Helpful? 0
  • +
  • -

#8 fisicx   User is offline

  • D.I.C Head

Reputation: 0
  • View blog
  • Posts: 57
  • Joined: 24-February 12

Re: Special Characters not recognised

Posted 07 May 2020 - 09:59 AM

@Ornstein - Yes, I use WP functions.

In this case it's just update_option('interest', $myarray);

No other functions, it's: call the CSV > fgetcsv > move to a new array > update_options

@CTphpnwb - UTF-8 all the way
Was This Post Helpful? 0
  • +
  • -

#9 ArtificialSoldier   User is offline

  • D.I.C Lover
  • member icon

Reputation: 2760
  • View blog
  • Posts: 8,064
  • Joined: 15-January 14

Re: Special Characters not recognised

Posted 07 May 2020 - 10:25 AM

It sounds like the file isn't being read like you think it is. Open the CSV with a text editor to verify it was saved correctly in the first place.
Was This Post Helpful? 1
  • +
  • -

#10 CTphpnwb   User is offline

  • D.I.C Lover
  • member icon

Reputation: 3837
  • View blog
  • Posts: 13,994
  • Joined: 08-August 08

Re: Special Characters not recognised

Posted 07 May 2020 - 11:41 AM

Check to see if that file has a BOM. If so, try saving it without one.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1