Exneus - JSON parser, generator and formatter for Elixir built on top of the new OTP json module

Your Libraries & OS Mentoring Libraries
, , , , , , ,
1

Exneus is an incredibly flexible and performant JSON parser, generator and formatter for Elixir. It is a wrapper for Elixir of Euneus, an Erlang library built on the top of the new OTP json module.

Both encoder and decoder fully conform to RFC 8259 and ECMA 404 standards and are tested using JSONTestSuite.

The new OTP json module was introduced in OTP-27, which is compatible with Elixir v1.17, but by installing json_polyfill, the minimum requirement reduces to OTP-24 and Elixir v1.12. I’ve not tested it in other versions, but the CI says it works.

Exneus/Euneus’s goal is to be very flexible and performant.

The first release of Exneus is an Erlangish lib. I just adapted Euneus’s Erlang code to Elixir. This will change :slight_smile:

Further explanation and examples are available at hexdocs.

Features

  • Encode
  • Decode
  • Stream
  • Minify
  • Format

Encode option details

  • codecs - Act like plugins. There are some built-in options, and it is also possible to create custom ones.

    Currently built-in encode codecs:

    • timestamp
    • datetime
    • ipv4
    • ipv6
    • records

  • codec_callback - - Overrides the default codec resolver.

  • nulls - Defines which values should be encoded as null.

  • skip_values - Defines which map values should be ignored. This option permits achieves the same behavior as Javascript, which ignores undefined values of objects.

  • key_to_binary - Overrides the default conversion of map keys to a string.

  • sort_keys - Defines if the object keys should be sorted.

  • keyword_lists - If true, converts keyword_lists into objects.

  • escape - Overrides the default string escaping.

  • encode_integer - Overrides the default integer encoder.

  • encode_float - Overrides the default float encoder.

  • encode_atom - Overrides the default atom encoder.

  • encode_list - Overrides the default list encoder.

  • encode_map - Overrides the default map encoder.

  • encode_tuple - Overrides the default tuple encoder.

  • encode_pid - Overrides the default pid encoder.

  • encode_port - Overrides the default port encoder.

  • encode_reference - Overrides the default reference encoder.

  • encode_term - Overrides the default encoder for unsupported terms, like functions.

Decode option details

  • codecs - Act like plugins. There are some built-in options, and it is also possible to create custom ones.

    Currently built-in decode codecs:

    • timestamp
    • datetime
    • ipv4
    • ipv6
    • pid
    • port
    • reference

  • null - Defines which term should be considered null.

  • binary_to_float - Overrides the default binary to float conversion.

  • binary_to_integer - Overrides the default binary to integer conversion.

  • array_start - Overrides the :json.array_start_fun/0 callback.

  • array_push - Overrides the :json.array_push_fun/0 callback.

  • array_finish - Overrides the :json.array_finish_fun/0 callback.

    In addition to the custom function, there are:

    • ordered
    • reversed

  • object_start - Overrides the :json.object_start_fun/0 callback.

  • object_keys - Transforms JSON objects key into Elixir term.

    In addition to the custom function, there are:

    • binary
    • copy
    • atom
    • existing_atom

  • object_push - Overrides the :json.object_push_fun/0` callback.

  • object_finish - Overrides the :json.object_finish_fun/0 callback.

    In addition to the custom function, there are:

    • map
    • keyword_list
    • reversed_keyword_list

Benchmark

The next step is to improve the performance since no optimization has been done yet. However, there is a simple benchmark comparing it with some known libraries, including the OTP json module.

IMPORTANT!

The benchmark result below just gives an idea of performance since the benchmark runs are very simplistic. Subsequent releases will update those values.

Also, to be an honest benchmark, all libs should produce the same output. I have not checked this.

Encode results:

Name                 ips
euneus           36.95 K
json (OTP)       35.85 K - 1.03x slower +0.83 μs
Exneus           35.71 K - 1.03x slower +0.94 μs
jiffy            34.70 K - 1.06x slower +1.76 μs
Jason            26.75 K - 1.38x slower +10.32 μs
thoas            16.40 K - 2.25x slower +33.92 μs
jsone            15.52 K - 2.38x slower +37.38 μs
jsx               4.21 K - 8.77x slower +210.29 μs
JSON              3.88 K - 9.52x slower +230.47 μs

Decode results:

Name                 ips
json (OTP)       18.79 K
euneus           18.06 K - 1.04x slower +2.16 μs
Exneus           17.68 K - 1.06x slower +3.34 μs
Jason            15.16 K - 1.24x slower +12.74 μs
jsone            13.82 K - 1.36x slower +19.17 μs
jiffy            12.76 K - 1.47x slower +25.15 μs
thoas            11.89 K - 1.58x slower +30.89 μs
jsx               4.93 K - 3.81x slower +149.64 μs
JSON              2.43 K - 7.74x slower +358.48 μs

If you liked those tools, find them helpful, and would like to see them improved, please consider sponsoring me or buying me some coffees :heart:

"Buy Me A Coffee"

Feel free to submit suggestions and to contribute :smiley:

https://hexdocs.pm/exneus/

4 Likes
2

Quick question, why erlang devs need Euneus and can’t use the json library directly from OTP?

3 Likes
3

As per the json EEP:

This EEP proposes a JSON library which:

  • should be easy to adopt in large codebases using one of the popular, existing, open-source JSON libraries;
  • will allow the existing open-source libraries with custom features (like support for Elixir protocols) to become thin wrappers around this library;
  • will improve, or at least not regress, performance compared to leading open-source JSON libraries.

The OTP json module is just a module with optional callbacks.

4

Huh? So the :json Erlang module was just an interface, not an actual implementation? And :euneus is the implementation?

5

Sorry, I wasn’t clear.

Using just the OTP :json module without any other lib is fine and is extremely fast! It provides encoding and decoding:

iex(1)> :json.encode(:foo)
[34, "foo", 34]
iex(2)> :json.decode("\"foo\"")
"foo"

However, the OTP :json module is available starting from OTP-27 and provides few options. On the other hand, :euneus is built on top of :json, providing more flexibility and options, including a JSON formatter.

For example, to decode Map keys as atoms:

iex(1)> :json.decode(~s({"foo":"bar"}), [], %{object_push: fn k, v, acc -> [{String.to_atom(k), v} | acc] end})
{%{foo: "bar"}, [], ""}
iex(2)> :euneus.decode(~s({"foo":"bar"}), %{object_keys: :atom})
%{foo: "bar"}

And that’s just one example. Also, I adapted the OTP json module into a lib called json_polyfill to increase the compatibility with OTP and Elixir versions, OTP-24 and Elixir v1.12, respectively.

Please let me know if this answers your question.

6 Likes
6

Yes it does, thank you.

1 Like
7

How does json_polyfill work? I was under the impression that the new :json module required some private C-level functions in OTP that weren’t available before

8

There is no C library involved. The OTP json module is a pure Erlang module.

EDIT

The json_polyfill just adapts some parts of the code to be compatible with other OTP versions.

2 Likes
9

Ah, I was specifically referring to this tweet about it, but hey if the tests pass, that says it works to me

1 Like
10

I know that @michalmuskala was involved with the creation of the new :json module, so I was expecting that Jason would eventually provide some sort of abstraction on top of that. Based on this assumption, it’s not clear to me what Exneus brings to the table that Jason would not (eventually I mean). AFAIK, the one thing missing from Jason is streaming support and for that, I guess most people use Jaxon - which also allows working with incomplete documents.

Don’t get me wrong, the project looks very cool and I think everyone can do whatever they want with their free time, but realisticly speaking I think it’s also important to check if there’s too much overlap so we can avoid “fragmentation”.

Anyways, it’s an interesting project, so it would be helpful to have a clearer comparison of the benefits Exneus provides over other libraries out there (especially considering that the author of Jason is also involved with the implementation behind your abstraction).

1 Like
11

Thiago, I truly appreciate your perspective and respect your comment. Not only for Elixir, but Michał is also an extremely important person to the Erlang community, and his contributions are always great!

I do not know exactly what you mean by “fragmentation.”. Don’t get me wrong too, but I’m not inclined to start a discussion here comparing Exneus to another existing library. The main points of Exneus are in the thread.

Each library has its strengths and weaknesses. If you find a library that addresses your project’s requirements, feel free to consider using it. At the end of the day, libraries are tools, and choosing one over another is entirely optional. Moreover, libraries can always learn from and benefit from one another, and I would be delighted if Exneus could bring value to others. :blush:

3 Likes