dimamik

Torus - Integrate PostgreSQL's search into Ecto queries

Torus is a plug-and-play Elixir library that seamlessly integrates PostgreSQL’s search into Ecto, streamlining the construction of advanced search queries.

The goal is to help developers create Ecto queries that return relevant results with optimal performance, and it’s best to show how on an example:

Pattern matching: Searches for a specific pattern in a string.

iex> insert_posts!(["Wand", "Magic wand", "Owl"])
...> Post
...> |> Torus.ilike([p], [p.title], "wan%")
...> |> select([p], p.title)
...> |> Repo.all()
["Wand"]

See like/5, ilike/5, and similar_to/5 for more details.

Similarity: Searches for items that are closely alike based on attributes, often using measures like cosine similarity or Euclidean distance. Is great for fuzzy searching and ignoring typos in short texts.

iex> insert_posts!(["Hogwarts Secrets", "Quidditch Fever", "Hogwart’s Secret"])
...> Post
...> |> Torus.similarity([p], [p.title], "hoggwarrds")
...> |> limit(2)
...> |> select([p], p.title)
...> |> Repo.all()
["Hogwarts Secrets", "Hogwart’s Secret"]

See similarity/5 for more details.

Text Search Vectors: Uses term-document matrix vectors for full-text search, enabling efficient querying and ranking based on term frequency. - PostgreSQL: Full Text Search. Is great for large datasets to quickly return relevant results.

   iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
   ...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
   ...> insert_post!(title: "Completely unrelated", body: "No magic here!")
   ...>  Post
   ...> |> Torus.full_text([p], [p.title, p.body], "uncov hogwar")
   ...> |> select([p], p.title)
   ...> |> Repo.all()
   ["Diagon Bombshell"]

See full_text/5 for more details.

Semantic Search: Understands the contextual meaning of queries to match and retrieve related content utilizing natural language processing. Read more about semantic search in Semantic search with Torus guide.

insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
insert_post!(title: "Completely unrelated", body: "No magic here!")

embedding_vector = Torus.to_vector("A magic school in the UK")

Post
|> Torus.semantic([p], p.embedding, embedding_vector)
|> select([p], p.title)
|> Repo.all()
["Diagon Bombshell"]

See semantic/5 for more details.

The above macros accept a list of options to customize their behavior. See function docs for examples. Most functions have an optimization section that might help you boost the performance of these search queries.

In upcoming plans, we’ll add support for highlighting search results and extend the search with hybrid search. Please let me know what do you think of it, and I’ll gladly hear any suggestions on how to make it better!

Most Liked

dimamik

BM25 Full-Text Search is now available via the new `Torus.bm25/5` macro

BM25 is a modern ranking algorithm that generally provides superior relevance scoring compared to traditional TF-IDF (used by full_text/5). This integration uses the pg_textsearch extension by Timescale.

See it in action on the demo page

Key features:

State-of-the-art BM25 ranking with configurable index parameters (k1, b)
Blazingly fast top-k queries via Block-Max WAND optimization (Torus.bm25/5 + limit)
Simple syntax: Post |> Torus.bm25([p], p.body, "search term") |> limit(10)
Score selection with :score_key and post-filtering with :score_threshold
Language/stemming configured at index creation via text_config

Requirements:

PostgreSQL 17+
pg_textsearch extension installed
BM25 index on the search column (with text_config for language)

See the BM25 Search Guide for detailed setup instructions and examples.

When to use bm25 vs full_text:

Use bm25/5 for fast single-column search with modern relevance ranking
Use full_text/5 for multi-column search with weights or when using stored tsvector columns

Special thanks to folks from Thinking Elixir for pointing to this fantastic pg_textsearch extension that does the heavy lifting here.

Post #18

dimamik

Hey!

Semantic search is finally here! Read more in Semantic search with Torus guide.
Shortly - it allows you to generate embeddings using a configurable (and chainable) adapters and use them to compare against the ones stored in your database.

Supported adapters (for now):

Torus.Embeddings.OpenAI - uses OpenAI’s API to generate embeddings.
Torus.Embeddings.HuggingFace - uses HuggingFace’s API to generate embeddings.
Torus.Embeddings.LocalNxServing - generate embeddings on your local machine using a variety of models available on Hugging Face
Torus.Embeddings.PostgresML - uses PostgreSQL PostgresML extension to generate embeddings
Torus.Embeddings.Batcher - a long‑running GenServer that collects individual embedding calls, groups them into a single batch, and forwards the batch to the configured embedding_module (any from the above or your custom one).
Torus.Embeddings.NebulexCache - a wrapper around Nebulex cache, allowing you to cache the embedding calls in memory, so you save the resources/cost of calling the embedding module multiple times for the same input.

And you can easily create your own adapter by implementing the Torus.Embedding behaviour.

So after you’ll pick your favorite embedding adapter, you can add semantic search:

def search(term) do
  search_vector = Torus.to_vector(term)

  Post
  |> Torus.semantic([p], p.embedding, search_vector, distance: :l2_distance, pre_filter: 0.7)
  |> Repo.all()
end

Future plans:

Release torus_example app that would allow experimenting with options and different search types to pick the best one
Add support for highlighting search results. (Base off of a ts_headline function)
Extend similarity search to support fuzzystrmatch extension distance options.

Links

Post #12

dimamik

Torus v0.5.2 is released!

New

New demo page where you can explore different search types and their options. It also includes semantic search, so if you’re hesitant - go check it out!
Other documentation improvements

Fixes

Correctly handles order: :none in Torus.semantic/5 search.
Updates Torus.Embeddings.HuggingFace to point to the updated feature extraction endpoint.
Suppresses warnings for missing ecto_sql dependency by adding it to the required dependencies. Most of us already had it, but now it’ll be explicit.
Correctly parses an array of integers in Torus.QueryInspector.substituted_sql/3 and Torus.QueryInspector.tap_substituted_sql/3. Now we should be able to handle all possible query variations.