Opened 5 years ago

Closed 5 years ago

#850 closed defect (fixed)

Encoding problems with JSON data

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: blocker Milestone: Reggie v4.1.1
Component: net.sf.basedb.reggie Keywords:


The Firefox update to Firefox 44.0.2 seems to have broken the encoding/decoding of JSON data that is posted between the browser and Reggie servlets. This was discovered by swedish letters åäö being stored as åäö.

Investigations showed that:

  • Reggie sets Content-Type: application/json request header when posting JSON data.
  • Firefox 43 automatically add UTF-8 to this: Content-Type: application/json; charset=UTF-8
  • Firefox 44 doesn't add anyting
  • If we don't set any Content-Type header at all, Firefox (both versions) automatically set: Content-Type: text/plain;charset=UTF-8
  • If a charset has been set, Tomcat uses that encoding but uses Latin1 otherwise. This is what caused the corruption of swedish letters.

There are several solutions:

  • We can let Reggie explicitly set charset=UTF-8 when posting JSON data.
  • We can explicitly tell Tomcat to use UTF-8 when parsing the JSON data.
  • Both of the above!

Change History (4)

comment:1 Changed 5 years ago by Nicklas Nordborg

Milestone: Reggie v4.xReggie v4.1.1
Status: newassigned

comment:2 Changed 5 years ago by Nicklas Nordborg

(In [3748]) References #850: Encoding problems with JSON data

Fixed on the server side by replacing all calls to HttpServletRequest.getReader() with a utility method in JsonUtil that instead uses HttpServletRequest.getInputStream() which is then wrapped by a Reader that is explicitely specifying 'UTF-8' encoding. In this way we do not have to rely on Tomcat to choose the encoding when converting from binary to text.

comment:3 Changed 5 years ago by Nicklas Nordborg

(In [3749]) References #850: Encoding problems with JSON data

Fixed on the browser side by explicitely setting Content-Type: application/json; charset=UTF-8 when sending JSON data.

Also removed code from a few places which created AJAX request objects that were never used.

comment:4 Changed 5 years ago by Nicklas Nordborg

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.