Dusty

Dusty

I am interested in implementing suffix trees in Elixir, with the primary goal of building a data structure that will allow fast autocomplete/autosuggest in LiveView. I see that there is an implementation of tries on Hex, but I think it’s important that the user not be required to match the string from the beginning. Given the string “Add New Account,” I would like the user to receive the suggestion for any substring (for example, “new,” or “acc”).

I have been reviewing this excellent article on using Ukkonen’s method to construct a suffix tree. There are some code examples on Rosetta Code, and I also found this article and this book chapter helpful.

At first, I thought it would be as simple as nesting linked lists. I had the idea that the algorithm could simply traverse the top level of the list until the beginning of the entered text was matched, and then move on to traversing the first nested list until further matches, and so on. It even appeared that the usual leaf marker, $, could be eliminated, because we could simply build up the matched string and return it when we arrived at [head | []].

However, further investigation has made me aware that I dramatically underestimated the complexity of constructing such a tree. Even using tuples to provide additional information ({:node, "substring"} or {:leaf, "substring"}), does not address the fact that Ukkonen’s method of construction requires links between nodes of the tree (meaning, jumping between nodes, rather than traversing a list in order).

Perhaps it isn’t possible to use existing Elixir data structures to create such a suffix tree, as the tree itself is a data structure with unique properties. In any event, I am interested in putting some time into this, but I really need to better frame the problem and the approach first.

The ultimate goal would be a Hex package with an API along these lines:

defmodule SuffixTree do
  @doc """
  Takes a list of strings, sorts them, and constructs a suffix tree.
  """
  def build(list, opts \\ []) do
    #...
  end

  @doc """
  Takes a user-entered substring and returns a list of matches from the given suffix tree.
  The number and sorting of matches are determined by opts.
  """
  def match(tree, substring, opts \\ [])
    #...
  end
end

My impression at this stage is that the building of the tree could be greatly simplified if it were done asynchronously and you didn’t care how long it took. The purpose of all the links between nodes in Ukkonen’s method appears to be a reduction in the time it takes to build the tree, from O(m2) to O(m) for a string of length m. Perhaps a full implementation of Ukkonen isn’t needed for smaller sets of strings.

How would you approach this? Am I missing some prior art that has already solved this in Elixir? Should I just hit the ElasticSearch docs and forget about this? Thanks in advance for any feedback.

Showing Posts 1 to 6

al2o3cr

al2o3cr

The core idea seems like :digraph from the Erlang stdlib could help, but that’s not likely to get comparable performance to the optimized algorithm.

Consider borrowing a technique from :digraph, however: it uses ETS tables to store vertexes / edges / etc mutably with random access. I used a similar-but-vastly-simpler approach for this Advent of Code problem - my solution uses an ETS table containing three-element tuples {marble, previous_marble, next_marble} to simulate a doubly-linked circular list.

Dusty

Dusty OP

Thanks! I will checkout :digraph. Bit by bit, I am getting closer to the heart of this. I found an implementation by Danny Yoo in Racket (the actual code is here), which is starting to look a bit more complete (despite my ignorance of Racket itself). In the references, there is a book by Dan Gusfeld that appears to be the best available reference on the matter.

Gusfeld’s explanation of the algorithm is probably the most detailed one out there, and Yoo’s implementation is enough to convince me that this should be possible in Elixir. I need to take the time to fully grasp Yoo’s code, but it seems like the apparent branching architecture of the tree can be simulated by using labels that indicate the parent and children of each node. Walking the tree is then possible by referencing these labels.

mindok

mindok

There’s also a pure elixir digraph implementation - GitHub - bitwalker/libgraph: A graph data structure library for Elixir projects · GitHub - performant and doesn’t use ETS

Dusty

Dusty OP

Awesome! Thanks for this.

atomkirk

atomkirk

did you manage to write suffix tree construction in elixir and do you mind sharing? :slight_smile:

Dusty

Dusty OP

Oh gosh, blast from the past.

The short answer is no, I never finished it. There are 2 main reasons:

  1. I migrated to using Rust primarily.
  2. The available implementations in Rust made it clear to me that performance is much better for suffix arrays, and most modern implementations use an optimization like that, rather than copying Ukkonen. For example, Andrew Gallant’s implementation, and specifically the articles referenced in his notes.

You can see the work I had done on the dev branch of this repo. It’s a long way from finished, but I had laid out a strategy (see ukkonen.md and the code comments).

— All posts loaded —

Where Next? Top

Trending in Questions Top

RSP87
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
nseaSeb
Hello, I know there is an approach for handling lists that allows for optimized traversal, but I can’t recall the specific method (somet...
New
RemyXRenard
I’m seeing that a list inside a Kino.DataTable will be interpreted as a charlist, even if the Kino.configure() is set to charlists: :as_l...
New
brecabral
Documentation While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
velrest
So my question is quite simple and i have found no conclusive answer on forum, google or AI. Should we use :erlang.float for Integer to ...
New
samoloth
Hi, I’ve just set up an application with ash_authentication. There is only magic link strategy for now, so there is no confirmation add o...
New
FlyingNoodle
If a change or preparation module uses Ash.Changeset.get_argument/2 or Ash.Query.get_argument/2 (or any of the other get_argument functio...
New

Other Trending Topics Top

mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
mhanberg
Hi everyone! The first release candidate for the Expert language server project is now available! We’ve published a press release detai...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews