Decide how to encode them. 
https://github.com/ufirstgroup/ymlr/pull/98
mruoss
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
Aren’t all Unicode “characters” printable by definition? What other encoding are you trying to convert them to?
0
mruoss
Maybe I’m using wrong terms. But a yaml encoder needs to encode different chars differently. For example: normal letters, digits or \n or \t can be left as they are. The bell character (\u0007) can be encoded as \u0007 or \a. Others need to be encoded in the form \u0033 or \x33.
0
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
If you want to produce “ascii only” yaml and escape everything else that’s fine, but it seems like yaml technically supports the full UTF8 spectrum. My overall point is that I would pick a well known subset of UTF8 like ASCII and then build your filter on top of that. All ASCII chars go in as is, non ascii are use the escaped utf8 form.
1
adamu
From looking at the YAML spec, it looks like:
Escape anything that has a C escape sequences to the form listed in the spec.
Then escape anything that’s explicitly not in the ranges the spec defines as printable with \u.
For each codepoint:
- In printable range? Leave it alone.
- Has c escape? Convert to that.
- Escape with \u
I think this is YAML-specific behaviour, so Elixir doesn’t have a default that does it, unless it’s provided by a YAML library, but it looks like that’s what you’re making 
[1] c-printable ::=
# 8 bit
x09 # Tab (\t)
| x0A # Line feed (LF \n)
| x0D # Carriage Return (CR \r)
| [x20-x7E] # Printable ASCII
# 16 bit
| x85 # Next Line (NEL)
| [xA0-xD7FF] # Basic Multilingual Plane (BMP)
| [xE000-xFFFD] # Additional Unicode Areas
| [x010000-x10FFFF] # 32 bit
0
mruoss
Yes, that’s almost exactly what I’m doing in the PR linked above. However, there are some things to consider. Let’s take char \u0085. It has an escape character in YAML (\N). But it is also in the c-printable range. So while theoretically it can be left alone, I preferred to encode it as \N because I think it is editor/reader friendlier. Would you agree? So my order is…
- Is \n or \t or? Leave it alone
- Has c escape? Convert to that.
- In printable range? Leave it alone.
- Escape with \u
Printing to console:
iex> IO.puts("a\u0085b")
a
b
:ok
Decode YAML (escape char inside double quotes) and printing to console:
iex> ~s("a\\Nb") |> YamlElixir.read_from_string!() |> IO.puts
a
b
:ok
Decode YAML (actual character) and printing to console:
iex> "a\u0085b" |> YamlElixir.read_from_string!() |> IO.puts
a
b
:ok
0