This problem space is pretty much an entire field of study, that of Natural Language Processing. Some techniques/prior art you could look at for jumping-off topics are TF-IDF, word2vec, and HuggingFace Tokenizers. The latter has a Hex package and is used internally by Bumblebee. As a casual bystander to the field, my understanding is this is pretty much the state of the art tactic right now, and I know that it is the kind of data representation used by LLMs.

You might combine tokenizers output with pgvector or another data store that can natively store and index vector data, which is the output of tokenizing. There’s a variety of techniques for comparing vectors but cosine similarity/distance seems to be one of the common suggestions. Pgvector for example can do this.

The Fly folks have a write up on a superficially similar ask, indexing and searching Hex packages:

If you have a finite, fixed set of labels to apply to each matching document, such as a short list of discrete topics, that might be doable as zero-shot classification with a pre-trained model.

Try this, embed everything and put it in Qdrant db then you can use filter parameter in search function

If you need help with Qdrant I wrote client for it (but it’s a bit outdated but should work tho) Just ping me or email me and I will help you with it.
https://github.com/marinac-dev/qdrant