ccgintranet

ccgintranet

Inserting Binary, Repo & Ecto

Hello everyone,

First post here and I am hoping someone can help expand my knowledge as I am clearly not handling a situation we are having properly.

We are using ExCrypto which directs to the underlying :crypto.block_encrypt/decrypt.

Our Ecto schema looks something like this:

    schema "bank" do
      field :uuid, :binary
      field :iv, :binary
      field :tag, :binary
    end

Part of the logic calls to ExCrypto.rand_bytes which returns a random binary stream.

An example here is if I attempt to insert it using Repo I will receive an expected struct that allows me to decrypt some cipher text.

%Api.Bank{
  __meta__: #Ecto.Schema.Metadata<:loaded, "bank">,
  id: 2,
  iv: <<155, 6, 136, 177, 177, 49, 143, 88, 129, 95, 129, 232, 197, 222, 16,
    200>>,
  tag: <<76, 5, 248, 99, 14, 234, 17, 201, 220, 233, 231, 86, 228, 193, 77,
    102>>,
  uuid: "a2eQg6kJiQ82CdpQD5SjBA=="
}

By requerying the database with that particular uuid (this would be a random string) I seem to get a completely different output.

%Api.Crypt{
  __meta__: #Ecto.Schema.Metadata<:loaded, "bank">,
  id: 2,
  iv: <<63, 6, 63, 63, 63, 49, 63, 88, 63, 95, 63, 63, 63, 63, 16>>,
  tag: <<76, 5, 63, 99, 14, 63, 17, 63, 63, 63, 63, 86, 63, 63, 77, 102>>,
  uuid: "a2eQg6kJiQ82CdpQD5SjBA=="
}

This tells me that the data being passed over to Repo changed somewhere between that call and it going into the database.

Working from this dataset in memory is fine, using ExCrypto.encrypt & ExCrypto.decrypt but as soon as I persist it to the database and retrieve it its garbled.

The reason its being persisted is we are using a backend RabbitMQ queue which picks up something we are doing at a later date.

My question, is am I not understanding how binaries work in Elixir or is something happening to my raw binary down the line when Repo inserts it?

Base.encode64 and Base.decode64 works fine, nothing is lost to and fro the database which makes me question how is the binary stream being handled here.

We use a MySQL database (5.6), InnoDB, utf8mb4_0900_ai_ci using BLOB columns.

Help is much appreciated,

Many thanks

Most Liked

voughtdq

voughtdq

It seems like PHP’s openssl_encrypt automatically encodes to base64. I wonder if you would have the same problem when using the OPENSSL_RAW_DATA option in your PHP implementation.

You may need to create a custom ecto type to encode/decode the binary.

voughtdq

voughtdq

<<155, 6, 136, 177, 177, 49, 143, 88, 129, 95, 129, 232, 197, 222, 16, 200>>
<<63, 6, 63, 63, 63, 49, 63, 88, 63, 95, 63, 63, 63, 63, 16>>

Something is changing it. If the byte is greater than 128 (with the current data we have), it seems to get dropped to 63.

NobbZ

NobbZ

128 and greater is not valid ASCII anymore, nore were the given string valid UTF8.
63 is ASCII (and UTF8) for a questionmark.

Can you please check the generated database query? Does that already contain the garbled data or is the query generated correctly? If you try to insert an opposing dataset manually into the database, containing only a single byte greater than 127, can you retrieve it back correctly?

Last Post!

ccgintranet

ccgintranet

Hello, apologies for the late reply.

Yes we are inserting the raw binary into the database from PHP’s end rather than base64.

Our PHP implementation and Elixir implemenation both share the same table in the database.

The data encrypted by PHPs end stores the IV and tag in the database and sends over the UUID to retrieve that same IV and tag from the database.

The same goes the other way and internally for both PHP and Elixir.

On the Elixir side we use ExCrypto to simply use that IV and tag to decrypt it; if the IV and tag were base64 ExCrypto would fail and our app wouldn’t function.

Thanks for the suggestion about the custom echo type. I will look into this!

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New

Other popular topics Top

joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 31525 112
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New