ConnorRigby

ConnorRigby

Nerves Core Team

Help implementing a notification system for Sqlite ecto adapter

So i’ve landed some features

In the lower level libs that make up the Sqlite.Ecto2 adapter. Now my problem is one of the actual implementation when working with Ecto. I know Ecto itself doesn’t support any notification system, but
the Postgrex adapter does support notifications. From skimming the source, i found that Postgrex just
opens another connection to the database which is something that sqlite can not support just because how it was built.

so one idea i had was to use Elixir.Registry, so every connection in the pool sends notifications out via Registry.

My issue is that sqlite sends a message that looks like
{:insert | :update | :delete, 'tablename', rowid}
(here are the docs for that)

Basically i get a rowid not a primary key or anything that is actually exposed to the Ecto Repo,
so even if an end user subscribed and got this message, they wouldn’t be able to easily lookup the data
that was affected because rowid is kind of an internal thing.

What i was thinking of doing is:

%{columns: c, rows: [row]} = Ecto.Adapters.SQL.query!(repo, "SELECT * FROM '#{table}' where rowid = #{rowid};", [])

And then sending that message to subscribers of the Registry, so a consumer would have to do:

receive do
   {action, 'users', %{columns: c, rows: [row]}} -> MyRepo.load(User, {result.columns, c, row})
end

The only problem i have with this is that every insert will also have a query to dispatch the notification,
and having user call load seems a little strange also.

First Post!

idi527

idi527

:waving_hand:

Thank you for implementing this.

But could the table name in the message be a binary instead of charlist?

If there isn’t a way to lookup the ecto schema module for a table name at runtime, I’d probably just leave the notification as is (that is, I’d just publish the rowid, without any preloads).

Also, can the nif “overload” the subscriber pid with messages? If the changes are published faster than the pid can handle them.

Most Liked

ConnorRigby

ConnorRigby

Nerves Core Team

unfortunately the implementation of the message dispatching is handled by a NIF, and the convention for that NIF is to use charlists, because the lib is an Erlang library, not Elixir. Charlists are also faster for NIFs if i remember correctly, since it doesn’t have to allocate, convert to term, deallocate etc.

I don’t think Ecto encourages looking up a schema module at runtime, since multiple schemas could use the same table.

I guess dispatching the rowid isn’t that bad. I could make a wrapper module that one could use by supplying a schema and translate it for the user.

ConnorRigby

ConnorRigby

Nerves Core Team

I don’t think its a private api, just part of how the Sqlite3 virtual machine works. You can read about rowid here. Basically it exists, and you can use it to index things, but you can’t look at how it works. (as far as my limited knowledge of the sqlite public api goes anyway)

In most cases the rowid is the primary key, (although this isn’t always true), the only problem is that sqlite internally doesn’t actually dispatch the callback (what sends data back to Beam) until after the row has been destroyed.

There actually is commit and rollback hooks (described here) but that would add more complexity to the NIF than i would ideally give it.

idi527

idi527

That would be perfect, at least for me, I think.

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

Other popular topics Top

bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 42716 114
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

We're in Beta

About us Mission Statement