Sounds like a bug if you’re getting caseclause error. Can you put an issue up with a repro?
Pegasus.Component is an internal tool that is used to help parse PEG grammars, those functions generate nimbleparsec definitions that are used internally by Pegasus.
I was messing around a bit more, here’s an incomplete example of json parsing with PEG (a few post_traverse functions are missing). If you don’t tag or collect, the data comes in reverse. When collected/tagged, it is a charlist. There are quite a few post_traverse functions in here.
Also, for the curious, it doesn’t come close to the performance of Jason. Haven’t had a chance to do any profiling, so not sure where the bottlenecks are, but let’s be honest, having a human-readable grammar for JSON is pretty incredible.
defmodule PegJSON do
require Pegasus
# https://www.json.org/json-en.html
# https://github.com/azatoth/PanPG/blob/master/grammars/JSON.peg
json_grammar = """
json_parser <- Value
Value <- S? ( Object / Array / String / True / False / Null / Number ) S?
Object <- ObjectStart
( ObjectPair ( Comma ObjectPair )*
/ S? )
ObjectEnd
ObjectStart <- "{"
ObjectEnd <- "}"
ObjectPair <- String ":" Value
Array <- "["
( Value ( "," Value )*
/ S? )
"]"
String <- S? ["] ( [^ " \ U+0000-U+001F ] / Escape )* ["] S?
Escape <- [\] ( [ " / \ b f n r t ] / UnicodeEscape )
UnicodeEscape <- "u" [0-9A-Fa-f] [0-9A-Fa-f] [0-9A-Fa-f] [0-9A-Fa-f]
True <- "true"
False <- "false"
Null <- "null"
Comma <- ","
Number <- Minus? IntegralPart FractionalPart? ExponentPart?
Minus <- "-"
#IntegralPart <- "0" / [1-9] [0-9]*
IntegralPart <- [0-9]*
FractionalPart <- "." [0-9]+
ExponentPart <- ( "e" / "E" ) ( "+" / "-" )? [0-9]+
S <- (' ' / '\t' / '\r\n' / '\n' / '\r')+
#S <- [ U+0009 U+000A U+000D U+0020 ]+
"""
json_parser_opts = [
json_parser: [export: true, parser: true],
S: [ignore: true],
Object: [tag: :object, post_traverse: {:post_object, []}],
ObjectPair: [post_traverse: {:post_obj_pair, []}],
ObjectStart: [ignore: true],
ObjectEnd: [ignore: true],
Array: [tag: :array],
#String: [tag: :str, post_traverse: {:post_str, []}],
String: [post_traverse: {:post_str, []}],
Escape: [tag: :escape],
Number: [tag: :number, post_traverse: {:post_number, []}],
Comma: [ignore: true],
#JSON: [tag: :json]
]
defp post_obj_pair(rest, args, context, _line, _offset) do
# IO.inspect rest
# IO.inspect context
[value, ":", key] = args
{rest, [ {key, value}], context}
end
defp post_str(rest, args, context, _line, _offset) do
#[str: str] = args
#IO.inspect args
#str = Enum.slice(args, 1..-2)|> Enum.reverse |> to_string
str = Enum.slice(args, 1..-2)|> to_string
#str = to_string(Enum.slice(args, 1..-2))
{rest, [ str ], context}
end
defp post_number(rest, args, context, _line, _offset) do
[number: str] = args
str = String.to_integer(to_string(str))
{rest, [ str ], context}
end
defp post_object(rest, args, context, _line, _offset) do
[object: object] = args
obj = Enum.into(object, %{})
{rest, [obj], context}
end
Pegasus.parser_from_string(json_grammar, json_parser_opts)
def bench do
t1 = "{\"a_key\":123,\"b_key\":456}"
t4 = "{\"a_key\": \"here\",\"b_key\": \"done\"}"
t3 = "{\"a\":1,\"b\":22}"
t2 = "{\"a\":1,\"b\": 2}"
Benchee.run( %{
"jason_decode" => fn -> Jason.decode(t1) end,
"peg_json_decode" => fn -> PegJSON.json_parser(t1) end,
},
print: [benchmarking: false, suite: false])
end
end
Thanks for the examples. Having seen this I went back to NimbleParsec docs — I didn’t pick up that post_traverse() needs to return {rest, List.t(), context}, and had it return a map instead. Maybe some Livebook tutorials may be helpful; I’ll see if I can build something after I’ve gotten more familiar.
Edit I had been parsing SVG path drawing instructions with very ugly regexes, with a series of
@path_extraction_QS ~r/(?<draw>[QqSs])((?<x1>[\d]+\.*[\d]*)+\s(?<y1>[\d]+\.*[\d]*)\s(?<x2>[\d]+\.*[\d]*)+\s(?<y3>[\d]+\.*[\d]*)+)+/
It’s totally worth the time to learn PEG / Pegasus.
defmodule Peg.SvgPath do
import NimbleParsec
require Pegasus
@svg_instruction_options [
Instructions: [tag: :instructions],
ThreeXY: [tag: :three_xy],
TwoXY: [tag: :two_xy],
OneXY: [tag: :one_xy],
Directional: [tag: :directional],
Close: [tag: :close],
XYPair: [tag: :xypair, post_traverse: :rename],
Number: [tag: :number, collect: true],
Space: [ignore: true]
]
Pegasus.parser_from_string(
"""
Instructions <- (ThreeXY / TwoXY / OneXY / Directional / Close)*
ThreeXY <- [Cc] Triple_XYPairs
TwoXY <- [QqSs] Double_XYPairs
OneXY <- [MmLlTt] XYPair
Directional <- [HhVv] Number
Close <- [Zz]
Triple_XYPairs <- XYPair Space XYPair Space XYPair
Double_XYPairs <- XYPair Space XYPair
XYPair <- Number Space Number
Number <- Integer (DecimalSeparator Integer)?
Integer <- [0-9]+
DecimalSeparator <- "."
Space <- ' '
""",
@svg_instruction_options
)
defparsec :get_instructions, parsec(:Instructions)
defp rename(rest, args, context, _line, _offset) do
[xypair: [number: [x], number: [y]]] = args
{rest, [%{x: Decimal.new(x), y: Decimal.new(y)}], context}
end
end
I’m trying to use this to parse CJK strings, and I’m at a loss whether it is possible to designate unicode strings. What I tried is:
defmodule ExCantonese.Markup.Parser.MVP do
import NimbleParsec
require Pegasus
def parse(markup) do
{:ok, [cjk: results], _remainder, _, _, _} = parse_markup(markup)
results
end
@cjk_options [
CJK: [tag: :cjk],
CJKchar: [tag: :cjk_char, collect: true]
]
Pegasus.parser_from_string(
"""
CJK <- CJKchar
CJKchar <- [a-z]
/ [\u3400-\u9FAF]
/ [\u20021-\u2F8A6]
""",
@cjk_options
)
defparsec :parse_markup, parsec(:CJK)
end
Against test cases of
cjk_2a = "我"
cjk_2b = "三"
cjk_3 = "a"
I expected a return for :cjk_char of ["我"], ["三"], and ["a"], but instead received:
[<<230>>][<<228>>]["a"]I think this is matching the char, but collecting only the first bit. Is this expected behaviour (in which case is it possible to work with non-alphanumeric ranges?)?
You’ll have to do a byte by byte match. The spec
Does not support \u.
You may want to check out how the zig parser handles “arbitrary Unicode”:
What is the syntax for matching byte?
I want to match – which is a special character with hex code 0x2013.
defmodule MyParser2 do
import NimbleParsec
require Pegasus
Pegasus.parser_from_string(
"""
header <- colon / special_minus
colon <- ":"
special_minus <- "0x2013"
""")
defparsec :parse_question, parsec(:header)
end
MyParser2.parse_question("–")
{:error, "expected string \"0x2013\"", "–", %{}, {1, 0}, 0}
Also, tried the following
special_minus <- 0x2013
special_minus <- [0x2013]
None of them compile.
