Pathex - a library for performing fast actions with nested data structures

Hi there,

I’ve been working on a Pathex library for quite a while now, and I’ve finally managed to create a release which is able to cover all requirements in nested structures access. Think of Pathex as Elixir’s Access but on steroids, or like Clojure’s Spectre but easier to learn and more efficient.

Tada,
https://hexdocs.pm/pathex/readme.html

Features

  1. Efficient in time. It’s 2-5 times faster than Access. HTML manipulations with Pathex are 5 times faster than Floki. To achieve this efficiency, Pathex generates pattern-matching cases at compile-time.

  2. Easy to use. Pathex is like a Map or Enum module, but for any structures. At it’s tiny core, it has everything you might need in your development needs. Errors are very descriptive. And I’ve put some effort into the documentation to be easy to navigate

  3. Functional. Pathex is built around functional lens idea, but it differs in some ways from it.

  4. Rich toolkit. Pathex works fine with lists, tuples, maps, structures, nested structures like AST or HTML. Pathex is reusable, because composition of two paths creates another path and so on.

Feedback

Feedback is really appreciated. I’d like to know what you’re thinking about library design because this library tries to extend the language!

80 Likes

Great project and documentation!

15 Likes

This looks great and I already have some use cases for it.

I like the design too - like the separation of the definition of the paths. I can see it would allow for cleaner code: separation of the definition of the structure of your data from the code that manipulates it.

1 Like

This looks awesome, great work!

I was looking at the Lenses docs and it wasn’t really clear to me the difference between matching and filtering (both the description and the examples are exactly the same)

Doc content in general is really good, but maybe some reordering could be helpful to clarify some things, e.g: cover the combinator syntax before the examples in the cheat sheet.

2 Likes

Thanks, I’ll a basic syntax in the Cheatsheet

The difference between matching and filtering is that
matching accepts pattern as a predicate. For example

iex> admin_lens = matching(%{role: :admin})
iex> Pathex.view(%User{name: "emoragaf", role: :admin}, admin_lens ~> path(:name))
{:ok, "emoragaf"}
iex> Pathex.view(%User{name: "hissssst", role: nil}, admin_lens ~> path(:name))
:error

will work only with values, which match the pattern %{role: :admin}.

And filtering lens supports any predicate as a function passed into it. For example,

iex> admin_lens = filtering(fn user -> user.role == :admin end)
iex> Pathex.view(%User{name: "emoragaf", role: :admin}, admin_lens ~> path(:name))
{:ok, "emoragaf"}
iex> Pathex.view(%User{name: "hissssst", role: nil}, admin_lens ~> path(:name))
:error

This lens does essentially the same as a matching lens above.

As you can see, every lens created with matching can be expressed with filtering, but matchingis easier to write, read and a is little bit faster than filtering

6 Likes

Loving this. I work with deep (up to 10 levels) sports stats data and this could be my ticket to code reduction. Thanks!

3 Likes

Spectre is/was a great idea, though it did not get the adoption it deserved. Not very easy to get started with.

Pathex seems to be an improvement - I was looking for something like that myself, because working with immutable nested structures is a pain.

2 Likes

This project is really great, and it’s gonna be a dependency for all of my future projects. I’m only worried that I might end up over using it. I kind of wish there was something like this in the standard library since nested data structures are so pervasive. And I really like the fact that you can throw maps, keyword list, and tuples, nested any which way without worrying about it.
In other words, thank you!

6 Likes

Just tweeted about it from theelixirbook account.

Great job, kudos.

1 Like

Released 2.1 just now, check out new features and fixes in changelog

1 Like

Very cool library :clap: I’m wondering what is the correct way to reduce some paths to a pathex at runtime?

For example I have this list of paths ["quiz", "questions", 0] and I want to create a pathex at runtime.

Now I’d do something like this:

["quiz", "questions", 0]
|> Enum.map(&Pathex.path/1)
|> Enum.reduce(&Pathex.concat(&2, &1))

Is there a way how I can have an empty pathex so I can reduce in one go?

Something like this:

["quiz", "questions", 0]
|> Enum.reduce(Pathex.empty(), &Pathex.concat(&2, Pathex.path(&1)))

Or is there even a better way?

1 Like

Hi, that’s a good question. First of all, I’d suggest to define as much paths as possible in compile time, because Pathex tries to optimize as much as possible in compile time. You can take a look at this doc to see the explanations about optimizations. The general idea is to provide constants as arguments, specify mods or annotate the path.

Otherwise, for creating paths in runtime you can refer to one of these examples

The most efficient way to do this is:

use Pathex
import Pathex.Lenses, only: [matching: 1]

case items do
  [head | tail] ->
    Enum.reduce(tail, path(head), fn right, left -> left ~> path(right) end)
  [] ->
    matching(_)
end

Or the shorter solution

use Pathex
import Pathex.Lenses, only: [matching: 1]

["quiz", "questions", 0]
|> Enum.reduce(matching(_), fn r, l -> l ~> r end)
1 Like

Thanks. Missed the matching/1

Do you think there’s a room for Pathex.new/1?

Thinking about:

def new(list \\ [])
  list
  |> Enum.reduce(matching(_), fn r, l -> l ~> r end)
end

I would be heppy to contribute. I get the compile time benefits, but sometimes it’s the way it is :man_shrugging:

1 Like

Yeah, that’s a good idea. I already have this in my TODO list, I’ll try to prioritize it for 2.2 or 2.3 versions then.

2 Likes

Hi, I’ve just released 2.2.0, you can now use this function:
https://hexdocs.pm/pathex/Pathex.Accessibility.html#from_list/2

5 Likes

Pathex 2.3 release is here :tada:

Nothing special, just

  1. Pathex.Short for shorter path definition. One can use just :x / : y instead of path :x / :y when using Pathex.Short
  2. Pathex.pattern for creating patterns from inlined paths. For example,
use Pathex, default_mod: :json
import Pathex

pattern(four, path :x / :y / :z / 3) = %{x: %{y: %{z: [1, 2, 3, 4]}}}
  1. Pathex now can be use-d inside functions or anything like this. This is now tested and supported.
  2. Paths inlining is now detected for aliased, imported and macro calls. This is done using Macro.Env.lookup* functions, therefore pathex is compatible only with elixir 1.13 or higher
  3. Spec fixes here and there, project formatting.

By the way, I’ve tested this release against gradient (from gradualizer project) and it seemed to be able to find some errors in specs, while failing to complete checking with exception and a bunch of false positives. And I’ve used mix_unused tool to find some unused functions (which helped me a lot)

4 Likes

Pathex 2.4.0 release

This release contains bug fixes and improvements in negative indexes for lists and tuples and improvements for force_* operations. For more information, refer to changelog

For example, force_setting a value in list used to result in something like

iex> force_set! [1, 2], path(4), 0
[1, 2, 0]

And now the empty space is filled with nil like this

iex> force_set! [1, 2], path(4), 0
[1, 2, nil, nil, 0]
3 Likes

Pathex 2.5.0 release

This release contains performance improvements, bug fixes and special functions for creating for_update-friendly lenses for structures and records. For more information, refer to changelog

For example,

iex> import Pathex.Accessibility
iex> uri_scheme_lens = from_struct(URI, :scheme)
iex> %URI{scheme: "http"} == Pathex.force_set(%{}, uri_scheme_lens, "http")
true

This is useful for doing deeply nested force_set and force_update operations

2 Likes