Hmm, I’m entirely mistaken, there are a couple remote calls to the module after compilation, like when it converts a string to an integer it calls NimbleParsec.__runtime_integer__/1 as one example of actually quite a few…
So… I don’t actually know? ^.^;
You have to tell the whole story here: it’s only as trivial to implement as long as all the prinitive parsers (even those that are user-defined). Remember the errors caused by a wrong implementarion of the symbols parser
Also, the integer parser is… extremely limited… It seems it does base10 only, no other options? o.O
Also it seems the integer parser is not capable of negative or positive definitions (which of course may be language dependent based on what you are parsing, thus not included by default in ExSpirit either since they are easy for the user to specify them), and yet it is called ‘integer’ instead of ‘unsigned_integer’ or ‘uint’ or so, the name is not quite accurate?
Hehe, fence errors! ^.^;
Honestly the old interface of ExSpirit is very low level, once I wrote defrule I should have rewritten, oh, most of it as defrule’s perhaps, but most of the parsers existed before defrule existed, sooo… >.>
Also another note, ex_spirit is built to, just like in C++'s version, handle more than just binaries, that just happens to be what the Text submodule focuses on, but the base parsers could all handle lists (like say of tokens) or a custom format or whatever as well. It doesn’t look like Nimble can support anything but binaries. Which is fine for me currently as I’ve not needed the other functionality yet (though it is a possibility in the future, hence why I left it open).
Also, is there a reason for the ‘literal’ thing? It’s very noisy when it seems that just a literal thing in code would be far easier to read (which is what my expression parser builder was going to support instead of needing lit(...) everywhere). I know it’s for easy piping, but a better wrapper around that would work as well.
Also parsec/1/2 creates a stacktrace entry, but so do the things like choice too. Also couldn’t it be optimized to call straight into the parser anyway, bypassing the caller function? 
Overall it looks cool, but it needs a couple changes, like integer being more properly named, handling more bases, a way to extend the parser easily, a way to handle state, byte handling, and a few dozen helper parser bits.
I see. We do keep a stack where we can store contexts internally, so if this is needed, we can support it. Just open up an issue with the API you have in mind and we can discuss it. I actually think the traverse combinator can be generalized into a context combinator.
Think about it like quoted expressions. You can compose and change as if they were runtime components but then you pass it to macros like def which compile them down. So while you are composing, you do need the library, but after you call defparsec you do not.
No, those calls happen at compilation time. The generated code doesn’t call NimbleParsec at all.
Not hard at all. I did not think about the use of bytes to extract offsets. I am wondering if there is a simpler schema that doesn’t require me to keep all of line, column and offset. There is also the issue that we can’t properly count UTF-8 columns when parsing codepoint by codepoint. So maybe I will keep the line as {line, last_line_byte} and the current_byte. Column can be properly be derived from those two.
Yes, the focus is on binaries because the goal is to explicitly leverage binary matching.
I agree. I will remove literal in favor of just passing the binary.
The problem with parsec/2 is not the call but the fact we need to handle the returned value and resume the callee. Also note choice/2 in nimble does not necessarily create a stacktrace entry if we know that all choices are bound patterns.
Thanks! integer will likely be kept as is but I will clarify the docs. And we don’t plan to handle more bases. As I said earlier, integer is built on top of existing primitives, so you can easily support other bases and make it as performant as the built-in integer.
A lot of my focus on the library was exactly in defining the low-level primitives, so I want to focus on capabilities rather than adding many different helpers. If we solve the former, the latter is simple and fast.
So I am very interested on constructs we don’t support yet so we can extract the underlying primitives out of them.
I think the “first argument” requirement is going away in 21
I don’t think you do, but that’s my fault. I haven’t explained myself well. From what I gather (haven’t had much time to read the source), you store a parser stack. That’s not remotely enough for what I’m talking about. To parse context-sensitive languages I need a place where I can store arbitrary data and retrieve or modify it later by a later parser. Your combinators seem to operate on tuples of the form:
{:ok, acc, stack, rest, line, column}
I’d need a tuple like this:
{:ok, acc, stack, state, rest, line, column}
And I’d need to get and set the state from inside my parsers. The simples example is XML, like @OvermindDL1, said, in which you need to match <x>...</x>. You need to keep a stack of XML tags, and that stack needs to be get and set by the user. The implementation in ExSpirit is a little more complex, because there is both a global state and a local state, but I think a global stats should be enough.
Also, for the reasons @OvermindDL1 and I outlined above, I’d like a byte_position value:
{:ok, acc, stack, rest, byte_position, line, column}
This starts to get a little hard to pattern match on, so you should give some serious thought to the idea of making it a record, which would act much closer to a struct and would be easier to pattern match on.
Because your combinators don’t have access to the full context, I don’t think you can support it without a major redesign. t’s totally ok if you want to only support parsing context-free languages, but if nimble is supposed to be comparable to ExSpirit it needs to give the users some access to the full context. Which, as I said above, makes it trivial to tag tokens with their source location (whatever your tokens are)
Ah I see, it returns AST without being a macro, cool.
Whooo!!
Ah true though that almost never the case in what I parse, the choices can be quite… complex… ^.^;
Also, is there anything that cancels backtracking so once something is parsed you know it must be that after or it is a failure (like my expect parser)? Without it, it is easy to get into some really bad worst-case parsing times with PEG parser.
Hmm, how might that look? I use a lot of bases at times and what base to use is calculated based on what was previously parsed (passed in via state) rather than being hardcoded variants.
State, adjusting state, passing state ‘as’ arguments to parsers, etc… That is the biggest one by far as without it some languages become extremely difficult to parse if not outright impossible to do correctly.
Very cool!
I really really like local state, makes things very simple. I don’t actually use global state at all and instead pass state ‘up’ via arguments and ‘down’ packed into results.
How would you maintain an indentation stack with only local state? You need all indent levels lower than the previous one. Would you push the stack into local state every time?