Improper Text Encoding (UTF-8)

[drexhacker]

Hello I have a REST Server returning JSON Responses using UTF-8 when I request the data through flutter_data I receive wrongly encoded text for example instead of getting agakuŋŋaanye go n’agayita as sent by the server, I get agakuÅÅaanye go nâagayita. I am using the defualt headers;

{
  "Content-Type": "application/json"
}

Really Need help here, everything works fine except this simple issue.

[drexhacker]

@frank06 I think the issue revolves back to UTF-8 charset for example my application supports various languages meaning it contains various characters in the UTF-8 encoding but maybe somehow somewhere they are limited. I found an almost similar issue in hivedb isar/hive#928 which I think somehow it relates to this.

[drexhacker]

Finally managed to solve the issue by modifying the sendRequest of the plugin locally on my Computer like below.

Original Version

...

    // response handling

    var contentType = 'application/json';

    try {
      if (response?.body.isNotEmpty ?? false) {
        contentType = response!.headers['content-type'] ?? contentType;
        final body = response.body;
        if (contentType.contains('json')) {
          responseBody = json.decode(body);
        } else {
          responseBody = response.body;
        }
      }
    } on FormatException catch (e, stack) {
      error = e;
      stackTrace = stack;
    }

...

Modified Version

...

    // response handling

    var contentType = 'application/json';

    try {
      if (response?.body.isNotEmpty ?? false) {
        contentType = response!.headers['content-type'] ?? contentType;
        if (contentType.contains('json')) {
          responseBody = json.decode(utf8.decode(response.bodyBytes));
        } else {
          responseBody = response.body;
        }
      }
    } on FormatException catch (e, stack) {
      error = e;
      stackTrace = stack;
    }

...

Hopefully it Helps Someone. But also take note that this for majorly UTF-8 Encoding, use with care.
@frank06 looking forward to your response on this, preferably a better and permanent solution.

[frank06]

Glad you found a workaround! I am hesitant to hardcode utf8 decoding so I'd need to analyze the situation. In tests, and probably if supplied by a backend, the header 'content-type': 'application/json; charset=utf-8' seems to make the response utf8 decoded whereas without it, the http package defaults to Latin1

[alexrothm]

Hey, I had the same issue. I should have read the workaround earlier... spent 2 hours trying to fix the encoding.

The JSON spec says that:

JSON text SHALL be encoded in Unicode. The default encoding is UTF-8. (RFC4627)

[frank06]

Thinking of changing application/json on the fly to application/json; charset=utf-8