Search engine without external database

lud

Hello,

I have a small project I will work on with a guy who is not a developer. He wants to be able to work on this tool by himself so I will kickstart the project and give him a dev environment. He will work on the client/javascript part.

I will write a tiny backend with Phoenix, and one important aspect of the project is to be able to type a search string in an input field and get a selection of matching object. For example if I type "table blu" I will get those matched names : "Small Blue turntables", "Big table", "Blurry thing", etc. If it is too complicated to implement it is ok to have to type *table to get "Small Blue turntables".

The search actually returns item IDs, not names.

That would be easy with PostgreSQL, here is the problem :

Connectivity may be limited sometimes so we want everything to work on a single laptop (no external tool like firebase).
His laptop is shared with its family, I would like to avoid installing Postgres or Docker.
The items table have a name field, but also another field for the name in French. Mybe that would be in another table if we want to add more languages. “table” is the same word in both languages so when I type a string, I have to search each word in both languages.
This setup is only for developing the app, if we make it public, I will have installed Postgres.
There will be around 10K items, definitely no more than 20K. When searching for an item, we will search within one of approx. 10 categories, so over 2000 items on average.

I know this sounds like making things complicated for no reason but I believe there is one way to make it run fast.

I was about to load everything in ETS or Mnesia, building a table per category and just walk over the table with a regex, but before I would like to know if someone here had a smart idea, because it will take time to build.
Also there are no table writes, item tables are static, maybe I could extract all unique words form all names, list all items for each of those words, and define static lookup functions with a macro. But regexes are not supported in guards so that would be limited to full and exact words.

Thanks for reading !

8 comments

#database #search-engine

0 1944 8

2020-02-27 16:06:31 UTC

Marked As Solved

Show Best Posts
Show All (oldest first)
Show All (newest first)

lucaong

This sounds like a great use-case for MiniSearch, a client-side full-text search engine written in JavaScript, small enough to run in the browser and with zero dependencies (disclaimer: I am the author of the library).

MiniSearch is routinely used for searching amongst tens of thousands of small items (e.g. all products in the catalog of a supplier), and the data can be indexed upon each page load (it usually takes sub-second, and can be done asynchronously).

Here’s a demo on a database of ~5000 songs.

Post #1

Also Liked

lud

It does not sound weird at all, it is actually nice. Data will be small since the browser will be fed with only id id/name/name_fr subset of the data, as there is no need to download the full items database.

And so instead of loading my data in tables I just have to build static lookup modules with macros.

Thank you !

Post #6