@josevalim would you be open to changing the “parser AST” in order to include metadata information that could e used with my combinators/1 macro?
Edit: typo
tmbb
mjadczak
I think that what I was thinking here was that in the case of C++, returning a parser “dynamically” is not extremely different from having a “static” one—all it is is a closure with the same code as for the “static” case, just with the compiler providing some bindings for some variables transparently and then populating those variables at runtime. That’s because all of the code is going through the same compiler—the C++ compiler.
In NimbleParsec’s case, the defparsec macro is in itself a compiler. Unless it’s somehow possible to move its logic higher up somehow (e.g. as a mix compiler) so that it can “transparently” act on code inside “dynamic” fns (i.e. inside closures), I think the “right” way to implement this is not to have worse-performing dynamic-style parsecs, but rather to empower NimbleParsec to also transform fn x, y, z -> parsec into a real piece of the optimised NimbleParsec code, but tracking all of the bindings to x, y and z and actually injecting them into the real generated code—essentially what the compiler does anyway when constructing closures.
josevalim
Creator of Elixir
Sure. Can you do a quick and small PR that changes one or two combinators, as an example? And once we agree on the shape we can change the rest.
tmbb
I thought it would be easy to make some minimal changes in NimbleParsec without understanding the whole codebase, but it hasn’t proved easy at all. I’ll have to take some time to understand it better before I can change anything.
mattste
Our frontend is implementing a custom language for users to query data (simpler than SQL). I am wondering if NimbleParsec has a way that could help me provide “autocomplete” like functionality.
For example, a user types if points and then an autocomplete would pop-up with a list of next possible tokens such as [">", "<", "="] .
tmbb
I’m trying to write a parser combinator that matches (and consumes) a regex at the beginning of the input string. For example:
defmodule RegexParserExample do
import NimbleParsec
import RegexParsec
# Parses javascript-style inline comments
comment = regex(~r[//(.*?)\n])
defparsec(:comment, comment)
end
The closest I have come is the following, using the post_traverse combinator:
defmodule RegexParsec do
import NimbleParsec
def regex(combinator \\ empty(), %Regex{} = re) do
anchored_re = Regex.compile!("^" <> re.source, re.opts)
post_traverse(combinator, {__MODULE__, :__regex__, [anchored_re]})
end
def __regex__(input, args, context, _line, _offset, re) do
case Regex.run(re, input) do
[bin | _] ->
length = byte_size(bin)
<< _part :: binary-size(length), _rest :: binary >> = input
# Doesn't consume input!
{[bin | args], context}
nil ->
{:error, "doesn't match regex #{inspect(re)}"}
end
end
end
The problem is that none of the *_traverse combinators allow me to to actually consume the input. They can only transform a result that comes from another parser, but not consume new bytes from the input string. The only way I can think of doing this is by having my parser return a parsec, but I’m not sure how to keep the context in a custom parsec (I don’t actually think any parser out there is actually using the context, but still).
I don’t know if even this possibility is recommended, because I don’t think there’s an API to add parsecs aside from the obvious one.
josevalim
Creator of Elixir
Can the new lookahead stuff be used for this? First you look ahead and then you parse if it matches.
tmbb
I can look ahead just fine, the problem is actually consuming the input. I’d need something like a combinator that would see how many bytes the regex matched and then advance that number of bytes in the string. I’ve actually found a solution that depends on creating a custom parsec from scratch, but it’s not the most elegant:
defmodule RegexParsec do
import NimbleParsec
defmacro defregexparsec(name, re) do
{evaluated_re, _} = Code.eval_quoted(re)
anchored_re = Regex.compile!("^" <> evaluated_re.source, evaluated_re.opts)
escaped_re = Macro.escape(anchored_re)
name__0 = String.to_atom(Atom.to_string(name) <> "__0")
quote do
def unquote(name)(input) do
unquote(name__0)(input, [], [], %{}, {1, 0}, 0)
end
def unquote(name__0)(input, acc, stack, context, {line, offset}, column) do
case Regex.run(unquote(escaped_re), input) do
[bin | _] ->
length = byte_size(bin)
# Split the string in two parts; keep the second part
<<_part::binary-size(length), rest::binary>> = input
# How many lines have we advanced?
lines = String.split("\n")
delta_lines = length(lines)
# The column is the byte offset of the last line (I think?)
new_column =
case lines do
[] -> 0
_ -> lines |> List.last() |> byte_size()
end
# I hope I'm returning the right format?
{:ok, [bin | acc], rest, context, {line + delta_lines, new_column}, column + length}
nil ->
{:error, unquote("expected to match regex #{inspect(evaluated_re)}"), input, context,
{line, offset}, column}
end
end
end
end
end
defmodule RegexParserExample do
import NimbleParsec
import RegexParsec
defregexparsec(:comment, ~r[//(.*?)\n])
defparsec(:comments, repeat(parsec(:comment)))
end
It depends on some implementation details, of nimble_parsec, but it seems to work.
tmbb
In case someone might actually think of using it, be aware that this approach is veeeery slow… Matching regexes in the middle of a parsec is a big “No” for performance. An interesting alternative would be to compile a regex to NimbleParsec combinators, but that seems more work than it’s worth.
joegiralt
Sorry to resurrect an old thread.
Is nimble parsec an appropriate tool for the frontend of a compiler? Or is it really just better to roll one’s own Lexer and parser, I understand that the latter give you the most freedom (and the most foot guns.)
For example, how does one idiomatically eliminate left recursion using this library?
Thanks!
Joe