FWIW, the byte 0xFA represents a “middle dot” character ( · ) on the old DOS code page 437.
Untangling old data like this is exceptionally difficult in general; I once had to migrate a database that had been storing whatever native character set was sent by clients (due to an old IE6 bug) and thus had everything from Arabic to Cyrillic ISO-8859 codepages mixed together. We had to hand-code a translation function from “customer country” to “likely character set” to deal with all that mess…
Hello, I have almost the same issue.
I’m trying to encode a 5 digit number with sha256 and set as a parameter in Guardian JTW token.
I get this error:
** (exit) an exception was raised:
** (Jason.EncodeError) invalid byte 0xE6 in <<39, 230, 80, 212, 221, 113, 28, 70, 32, 224, 187, 31, 151, 119, 135, 64, 98, 182, 156, 148, 203, 167, 9, 232, 6, 132, 32, 193, 154, 195, 82, 14, 255, 75, 121, 167, 68, 155, 55, 111, 108, 62, 12, 85, 98, 82, 225, 222, 11, 147, …>>
(jason 1.2.2) lib/jason.ex:150: Jason.encode!/2
(jose 1.11.1) src/jwt/jose_jwt.erl:89: :jose_jwt.to_binary/1
(jose 1.11.1) src/jwt/jose_jwt.erl:172: :jose_jwt.sign/3
(guardian 2.1.1) lib/guardian/token/jwt.ex:259: Guardian.Token.Jwt.create_token/3
(guardian 2.1.1) lib/guardian.ex:765: Guardian.returning_tuple/1
(guardian 2.1.1) lib/guardian.ex:594: Guardian.encode_and_sign/4
(guardian 2.1.1) lib/guardian/plug.ex:204: Guardian.Plug.sign_in/5
After I added : encrypted = String.chunk(encrypted, :valid) the I get this error:
Request: POST /api/user/authenticate/mobile
** (exit) an exception was raised:
** (Jason.EncodeError) invalid byte 0xAC in <<172, 255>>
(jason 1.2.2) lib/jason.ex:150: Jason.encode!/2
(jose 1.11.1) src/jwt/jose_jwt.erl:89: :jose_jwt.to_binary/1
(jose 1.11.1) src/jwt/jose_jwt.erl:172: :jose_jwt.sign/3
(guardian 2.1.1) lib/guardian/token/jwt.ex:259: Guardian.Token.Jwt.create_token/3
(guardian 2.1.1) lib/guardian.ex:765: Guardian.returning_tuple/1
(guardian 2.1.1) lib/guardian.ex:594: Guardian.encode_and_sign/4
(guardian 2.1.1) lib/guardian/plug.ex:204: Guardian.Plug.sign_in/5
Many invalid chars are gone but Looks like still there are some invalid chars remaining.
What do you recommend?
Two things:
String.chunk returns a list with two binaries (“valid” and “invalid” characters), but this passes them BOTH along to be serialized to JSON, thus the errorThe values in JSON need to be valid UTF-8; if you’re trying to pass arbitrary binary data, consider using an encoding format like hexadecimal (34f89e4c23a0d) or base64. Elixir supports them through the Base module in stdlib.