warmwaffles

warmwaffles

Need help testing and working through an ecto3 and sqlite adapter

During the winter storm here in Texas, I was couped up without reliable electricity so I was bored and started working on an sqlite3 nif and ecto3 adapter.

I had seen esqlite and sqlite_ecto2 and thought I would try my hand at just doing it all in elixir.

My goal with the project was to make the sqlite3 adapter as configurable as possible. Including changing cache pragma directives, full text search etc…

Anyways, I’m looking for some help knocking out bugs and finding them. I am currently using them on a personal project of mine until I can get this stable.

https://github.com/warmwaffles/exqlite

Most Liked

kevinlang

kevinlang

The core driver and its adapter are coming along nicely. Only a couple more tests left to onboard of the entire ecto and ecto_sql integration test suite.

I ran some benchmarks for exqlite comparing it to postgrex and myxql and found that it does much better than I expect, having up to 10x speed increase for inserts, for instance. This is far more than I expected, to the point where I think I may have ran something wrong, but it looks like everything is set up correctly.

https://github.com/warmwaffles/exqlite/pull/111

Exciting!

Eiji

Eiji

I’m especially interested using sqlite3 in my apps. I have tested it on myself:

ecto_example.exs
Mix.install([:ecto_sql, {:exqlite, github: "warmwaffles/exqlite"}])

defmodule Repo do
  use Ecto.Repo, adapter: Ecto.Adapters.Exqlite, otp_app: :my_app
end

defmodule Migration do
  use Ecto.Migration

  def change do
    create table("todo_lists") do
      add(:title, :string)
      timestamps()
    end

    create table("todo_items") do
      add(:description, :string)
      add(:type, :string)
      timestamps()
    end

    create table("todo_list_items") do
      add(:todo_item_id, references(:todo_items))
      add(:todo_list_id, references(:todo_lists))
    end
  end
end

defmodule TodoList do
  use Ecto.Schema

  schema "todo_lists" do
    field(:title)
    many_to_many(:todo_items, TodoItem, join_through: TodoListItem)
    timestamps()
  end
end

defmodule TodoListItem do
  use Ecto.Schema

  schema "todo_list_items" do
    belongs_to(:todo_list, TodoList)
    belongs_to(:todo_item, TodoItem)
  end
end

defmodule TodoItem do
  use Ecto.Schema

  schema "todo_items" do
    field(:description, :string)
    field(:type, :string)
    timestamps()
  end
end

defmodule Example do
  alias Ecto.Query
  require Query

  def cleanup do
    Repo.stop()
  end

  def prepare do
    Application.put_env(:my_app, Repo,
      database: "example",
      pool_size: 10,
      show_sensitive_data_on_connection_error: true,
      username: System.get_env("USER")
    )

    Application.ensure_all_started(:ecto_sql)
    Application.ensure_all_started(:postgrex)
    Repo.__adapter__().storage_down(Repo.config())
    Repo.__adapter__().storage_up(Repo.config())
    Repo.start_link()
    Ecto.Migrator.up(Repo, 1, Migration)
  end

  def sample do
    TodoList
    |> Query.from(as: :todo_list)
    |> Query.join(:inner, [todo_list: todo_list], assoc(todo_list, :todo_items), as: :todo_items)
    |> Query.where([todo_items: todo_items], todo_items.type != "hobby")
    |> Query.preload([todo_list: todo_list, todo_items: todo_items], todo_items: todo_items)
    |> Repo.all()
    |> IO.inspect()
  end

  def seed do
    Repo.insert(%TodoList{
      title: "Hobby example",
      todo_items: [%{description: "hobby1", type: "hobby"}]
    })

    Repo.insert(%TodoList{
      title: "Job example",
      todo_items: [%{description: "job1", type: "job"}, %{description: "job2", type: "job"}]
    })
  end
end

Example.prepare()
Example.seed()
Example.sample()
Example.cleanup()
elixir ecto_example.exs
* Getting exqlite (https://github.com/warmwaffles/exqlite.git)
remote: Enumerating objects: 77, done.        
remote: Counting objects: 100% (77/77), done.        
remote: Compressing objects: 100% (62/62), done.        
remote: Total 802 (delta 23), reused 55 (delta 12), pack-reused 725        
origin/HEAD set to main
Resolving Hex dependencies...
Dependency resolution completed:
New:
  connection 1.1.0
  db_connection 2.3.1
  decimal 2.0.0
  ecto 3.5.8
  ecto_sql 3.5.4
  elixir_make 0.6.2
  telemetry 0.4.2
* Getting ecto_sql (Hex package)
* Getting db_connection (Hex package)
* Getting decimal (Hex package)
* Getting ecto (Hex package)
* Getting elixir_make (Hex package)
* Getting telemetry (Hex package)
* Getting connection (Hex package)
==> connection
Compiling 1 file (.ex)
Generated connection app
===> Compiling telemetry
==> decimal
Compiling 4 files (.ex)
Generated decimal app
==> elixir_make
Compiling 1 file (.ex)
Generated elixir_make app
==> db_connection
Compiling 14 files (.ex)
Generated db_connection app
==> ecto
Compiling 56 files (.ex)
warning: Decimal.cmp/2 is deprecated. Use compare/2 instead
  lib/ecto/changeset.ex:2160: Ecto.Changeset.validate_number/6

Generated ecto app
==> ecto_sql
Compiling 26 files (.ex)
Generated ecto_sql app
==> exqlite
mkdir -p /tmp/mix_installs/elixir-1.12.0-dev-erts-12.0/5f38105e26a7022da3e48e89d9b11bd6/_build/dev/lib/exqlite/priv
cc -g -O3 -Wall -I"/home/…/.asdf/installs/erlang/24.0-rc1/erts-12.0/include" -Isqlite3 -Ic_src -DSQLITE_THREADSAFE=1 -DSQLITE_USE_URI -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_GEOPOLY -DSQLITE_OMIT_DEPRECATED -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_RBU -DNDEBUG=1 -shared -fPIC -fvisibility=hidden -Wl,-soname,libsqlite3.so.0 sqlite3/sqlite3.c c_src/sqlite3_nif.c -o /tmp/mix_installs/elixir-1.12.0-dev-erts-12.0/5f38105e26a7022da3e48e89d9b11bd6/_build/dev/lib/exqlite/priv/sqlite3_nif.so
Compiling 13 files (.ex)
warning: unused import String
  lib/ecto/adapters/exqlite.ex:5

Generated exqlite app

04:38:54.636 [info]  == Running 1 Migration.change/0 forward

04:38:54.645 [info]  create table todo_lists

04:38:54.648 [info]  create table todo_items

04:38:54.650 [info]  create table todo_list_items

04:38:54.654 [info]  == Migrated 1 in 0.0s

04:38:54.792 [debug] QUERY OK db=0.0ms idle=182.7ms
begin []

04:38:54.795 [debug] QUERY OK db=0.8ms
INSERT INTO todo_lists (title,inserted_at,updated_at) VALUES (?,?,?) ["Hobby example", "2021-03-01T03:38:54", "2021-03-01T03:38:54"]

04:38:54.795 [debug] QUERY OK db=0.6ms
INSERT INTO todo_items (description,type,inserted_at,updated_at) VALUES (?,?,?,?) ["hobby1", "hobby", "2021-03-01T03:38:54", "2021-03-01T03:38:54"]

04:38:54.796 [debug] QUERY OK db=0.1ms
INSERT INTO todo_list_items (todo_item_id,todo_list_id) VALUES (?,?) [1, 1]

04:38:54.798 [debug] QUERY OK db=1.9ms
commit []

04:38:54.798 [debug] QUERY OK db=0.0ms idle=194.0ms
begin []

04:38:54.799 [debug] QUERY OK db=0.4ms
INSERT INTO todo_lists (title,inserted_at,updated_at) VALUES (?,?,?) ["Job example", "2021-03-01T03:38:54", "2021-03-01T03:38:54"]

04:38:54.799 [debug] QUERY OK db=0.1ms
INSERT INTO todo_items (description,type,inserted_at,updated_at) VALUES (?,?,?,?) ["job1", "job", "2021-03-01T03:38:54", "2021-03-01T03:38:54"]

04:38:54.799 [debug] QUERY OK db=0.1ms
INSERT INTO todo_list_items (todo_item_id,todo_list_id) VALUES (?,?) [2, 2]

04:38:54.799 [debug] QUERY OK db=0.0ms
INSERT INTO todo_items (description,type,inserted_at,updated_at) VALUES (?,?,?,?) ["job2", "job", "2021-03-01T03:38:54", "2021-03-01T03:38:54"]

04:38:54.799 [debug] QUERY OK db=0.0ms
INSERT INTO todo_list_items (todo_item_id,todo_list_id) VALUES (?,?) [3, 2]

04:38:54.801 [debug] QUERY OK db=1.7ms
commit []

04:38:54.804 [debug] QUERY OK source="todo_lists" db=0.1ms queue=0.3ms idle=199.6ms
SELECT t0.id, t0.title, t0.inserted_at, t0.updated_at, t1.id, t1.description, t1.type, t1.inserted_at, t1.updated_at FROM todo_lists AS t0 INNER JOIN todo_list_items AS t2 ON t2.todo_list_id = t0.id INNER JOIN todo_items AS t1 ON t2.todo_item_id = t1.id WHERE (t1.type != 'hobby') []
[
  %TodoList{
    __meta__: #Ecto.Schema.Metadata<:loaded, "todo_lists">,
    id: 2,
    inserted_at: ~N[2021-03-01 03:38:54],
    title: "Job example",
    todo_items: [
      %TodoItem{
        __meta__: #Ecto.Schema.Metadata<:loaded, "todo_items">,
        description: "job1",
        id: 2,
        inserted_at: ~N[2021-03-01 03:38:54],
        type: "job",
        updated_at: ~N[2021-03-01 03:38:54]
      },
      %TodoItem{
        __meta__: #Ecto.Schema.Metadata<:loaded, "todo_items">,
        description: "job2",
        id: 3,
        inserted_at: ~N[2021-03-01 03:38:54],
        type: "job",
        updated_at: ~N[2021-03-01 03:38:54]
      }
    ],
    updated_at: ~N[2021-03-01 03:38:54]
  }
]

Many thanks! :heart:

warmwaffles

warmwaffles

No problem man. I have the same desire too. I wanted to try a quick poor mans map reduce with multiple sqlite databases. Thought it would be something cool to demonstrate and write about.

If you run into ANY bugs. Please report them in the issues. I am really trying to push for stability and then possibly push to get the adapter portion pulled into ecto_sql.

Where Next?

Popular in Discussions Top

PragTob
Hello everyone, I know we had quite some threads (read through lots of them) about background job processing but it remains a hotly deba...
New
fireproofsocks
This is more of a general question, but I’m wondering how other people in the community think about the pattern matching in function sign...
New
chuck
Let me start by stating an assumption: Phoenix is a great approach to building REST APIs. There are many reasons for this, but I will ass...
New
nburkley
AWS re:Invent is on at the moment with some interesting announcements. One new feature in particular is the Lambda Runtime API for AWS La...
New
tmbb
This is a post to discuss the new Phoenix LiveView functionality. From Chris’s talk, it appears that they generate all HTML on the serve...
342 18146 126
New
rower687
Hi all, I’ve been reading a lot about the “let it crash” term and how supervising processes and the whole messaging passing make an elixi...
New
eteeselink
Hi all, In the last days, two things happened: A blog post titled “They might never tell you it’s broken” made the rounds. It’s about ...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
chulkilee
Here are the list of HTTP client libraries/wrappers, and some thoughts on HTTP client in general. I’d like to hear from others how they w...
New
acrolink
How does the two languages compare when it comes to server side application development? Any experiences or ideas? Thank you.
New

Other popular topics Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52341 488
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 126479 1222
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement