The answer came from Frerich on the elixir slack.

I’d gotten too far into think about the parsing aspect of the problem and didn’t see the purely functional aspect - eager vs lazy evaluation.

The solution in the end is simple, introduce an intermediate parser that sits between value() and list()

My first use of a macro in Elixir:

defmacro lazy(parser) do
    quote do
      Parser.new(
        :lazy,
        fn ctx -> Parser.invoke(unquote(parser), ctx) end
      )
    end
  end

Now I can write:

def value() do
  choice([
    number_value(),
    boolean_value(),
    string_value(),
    lazy(list_value())
  ])
)

And the recursion is broken.

Congradulations on writing your first macro!

Indeed, this is one of the places where lazy evaluation makes an algorithm simpler.
Situations like this, where infinite recursion happens while building an intermediate structure for things that ‘may’ exist to an unknown (but finite) depth, come up from time to time.

The problem definitely is related to left-recursion. One could see it as left-recursion happening already before starting the parsing process itself.

In this particular example, you might be able to get away with a definition of lazy as a function (rather than a macro) like this:

def lazy(parser_producing_fun) do
  Parser.new(:lazy, fn cx -> Parser.invoke(parser_producing_fun.(), ctx) end
end

However, you’d need to call it as lazy(&list_value/0 instead of lazy(list_value()) so it alters the way it has to be used a little. Your choice

I recently finally took the time to watch that video myself.

I am wondering if I’m confusing stuff when I’m thinking that Saśa Jurić touches on the topic in the video already - starting at around 49:57min he talks about the problem of a subquery being its own select_statement() parser function.

He ends up evaluating the term lazily, just like you ended up doing, so I think it is the same situation you were dealing with?