Encoding problems with JSON data

[Nicklas Nordborg]

The Firefox update to Firefox 44.0.2 seems to have broken the encoding/decoding of JSON data that is posted between the browser and Reggie servlets. This was discovered by swedish letters åäö being stored as åäö.

Investigations showed that:

• Reggie sets Content-Type: application/json request header when posting JSON data.
• Firefox 43 automatically add UTF-8 to this: Content-Type: application/json; charset=UTF-8
• Firefox 44 doesn't add anyting
• If we don't set any Content-Type header at all, Firefox (both versions) automatically set: Content-Type: text/plain;charset=UTF-8
• If a charset has been set, Tomcat uses that encoding but uses Latin1 otherwise. This is what caused the corruption of swedish letters.

There are several solutions:

• We can let Reggie explicitly set charset=UTF-8 when posting JSON data.
• We can explicitly tell Tomcat to use UTF-8 when parsing the JSON data.
• Both of the above!

[Nicklas Nordborg]

(In [3748]) References #850: Encoding problems with JSON data

Fixed on the server side by replacing all calls to HttpServletRequest.getReader() with a utility method in JsonUtil that instead uses HttpServletRequest.getInputStream() which is then wrapped by a Reader that is explicitely specifying 'UTF-8' encoding. In this way we do not have to rely on Tomcat to choose the encoding when converting from binary to text.

[Nicklas Nordborg]

(In [3749]) References #850: Encoding problems with JSON data

Fixed on the browser side by explicitely setting Content-Type: application/json; charset=UTF-8 when sending JSON data.

Also removed code from a few places which created AJAX request objects that were never used.