How to serialize/unserialize elixir map data for RabbitMQ?

Questions / Help
1

I’ve been working on a RabbitMQ publisher and discovered that you can only publish strings to a RabbitMQ queue. The problem is I’m trying to publish a map to the RabbitMQ queue and it won’t let me.

For example:

my_map = %{message: "test", userid: 234}

I tried using Jason.encode!(my_map) to serialize the map before publishing it to the queue, which seems to work. But when I receive the message out of the queue in a later part of the app, I use Jason.decode!(my_map_from_queue), and I get this:

my_map = %{"message" => "test", "userid" => 234}

I also considered:

:erlang.term_to_binary(my_map)
:erlang.binary_to_term(my_map)

However, this thread mentioned it is unsafe to do on untrusted data: Serialize and unserialize structs, tuples, keyword lists, atoms? - #5 by sezaru

My intention is to save messages from a chat room to the database, so the message text being submitted could be anything and untrusted input coming from the chat room.

The original map being submitted is pre-formatted to a specific database schema. The goal was to gather all chat messages to RabbitMQ and use a producer to collect these messages in batches and use Repo.insert_all([list of maps]). The hope was that anything going into RabbitMQ would come out exactly the same.

What is a good solution for the serialization or “stringifying” and “destringifying” things going into and out of RabbitMQ queues?

2

What we do at my current employer is using MessagePack through the Msgpax library. Works well and keeps the door open for having a multi-language stack, if we ever need it.

3 Likes
3

Hello,

you can use keys: :atoms option for Jason.decode:

iex> my_map = %{message: "test", userid: 234}
%{message: "test", userid: 234}
iex> json = Jason.encode!(my_map)            
"{\"message\":\"test\",\"userid\":234}"
iex> Jason.decode!(json, keys: :atoms)
%{message: "test", userid: 234}
4

Hey @myefomit welcome. Unfortunately I don’t think this is good advice here. Decoding to atoms will create new atoms if the message keys change. Unless the shape of the RabbitMQ messages never changes this will eventually lead to excessive memory use and crash the application.

2 Likes
5

I think you can use binary_to_term/2 with the safe option.

:erlang.binary_to_term(my_map, [:safe]).

this only use atoms that have been created earlier, but you have to deal with the possible exception raising.

4 Likes
6

As a bonus you won’t ever get attacked by serialized functions inside the payload. AFAIR the :safe flag does not include functions.

2 Likes