UTF-8 Solr answer yields charlist

Currently I have a problem with an answer I’m getting from my Solr server.

When I request it directly from JavaScript or with Postman/Insomnia I get an UTF-8 document that displays my umlauts correctly. E. g. one value contains the german word Tür which will be rendered as is.

When I make the request from Elixir I’ll get a charlist as the documents body. This is then parsed as JSON and theoretically working as expected, except that now the Tür gets rendered as Tür.

What am I doing wrong here? I already tried converting it to string, which only yields a bitstring.

Could you give us a minimal charlist of the document body? Which JSON library are you using?
And how rendering works?

Are you supplying the result Tür to be rendered in an HTML page and it is getting broken?

If Yes You can check the encoding used in the html document that will instruct the browser to use the correct one: https://www.w3schools.com/tags/att_meta_charset.asp

I’ll try to produce a minimal example.

Which JSON library are you using?

Jason

And how rendering works?

The answer is rendered as JSON via Absinthe. I have a struct where I have annotated which fields should be rendered by Jason which I then feed to Absinthe.

But I think the real problem occurs before actually rendering it since the string is already sent as a charlist in the response of the Hui Solr library.


Now that I wrote it, I should probably try to send the request via HTTPoison directly or try another library like mint in order to check if it’s a transport problem…

I have checked the Hui library and it defaults to httpc as http client, but since you mentioned HTTPoison I assume you have configured it as the http client, otherwise since httpc is erlang it could be causing the encoding problem, and the behaviour of returning a charlist as the response.

From their documentation you can configure like:

config :hui, :default
      http_client: Hui.Http.Httpoison
1 Like

The problem is indeed httpc. Httpoison had no effect when I first tried it, because the module was missing and it silently failed.

That’s why I assumed a deeper issue… Thanks for the pointer! :sweat_smile:

1 Like