Parsing custom query language and query Ecto database

Yes, shift-reduce errors can be quite difficult to find and fix. From the very start I sort of dived in at the deep end by implementing leex , and yecc is actually a very old erlang tool.

but I’m wondering … should I be scared of the quoted format. I’m not sure why I should not just produce it from my parser and put it into a macro?

iex(162)> quote do
...(162)> from c in City, where: (c.country == "Sweden") or
...(162)>                        (c.country == "USA" and c.name == "New York")
...(162)> end
{:from, [context: Elixir, import: Ecto.Query],
 [{:in, [context: Elixir, import: Kernel],
   [{:c, [], Elixir}, {:__aliases__, [alias: false], [:City]}]},
  [where: {:or, [context: Elixir, import: Kernel],
           [{:==, [context: Elixir, import: Kernel],
             [{{:., [], [{:c, [], Elixir}, :country]}, [], []}, "Sweden"]},
            {:and, [context: Elixir, import: Kernel],
             [{:==, [context: Elixir, import: Kernel],
               [{{:., [], [{:c, [], Elixir}, :country]}, [], []}, "USA"]},
              {:==, [context: Elixir, import: Kernel],
               [{{:., [], [{:c, [], Elixir}, :name]}, [], []}, "New York"]}
             ]}
           ]}
  ]
 ]}
iex(164)> "city where country='Sweden' or (country='USA' and name='New York')" |>
...(164)> to_charlist() |> :lexer.string() |>                           
...(164)> (fn {_, x, _} -> x end).() |> :parser.parse() |>              
...(164)> (fn {_, x} -> x end).()
{:where, {:domain, 'city'}, 
 {:atom_or, 
  {{:operator, :cmp_eq}, ['country'], 'Sweden'},
  {:atom_and, 
   {{:operator, :cmp_eq}, ['country'], 'USA'},
   {{:operator, :cmp_eq}, ['name'], 'New York'}}}}

Currently forage only supports intersection, not union. It’s easy to add support for unions, though.

Well my first unix was pre-System III and it’s friends lex and yacc. I think they qualify as ancient

And thanks for writing them as part of Erlang, it feels like they should belong in any build system and it’s great they are standard issue.

Yes, pre-System III lex and yacc qualify as ancient

While I did implement leex, I cannot take credit for yecc. The first yecc versions were implemented by another guy at the Ericsson Computer Science Lab, Carl Wilhelm Welin.

since I’m here to learn, I would like to go through this macro idea.

I met two difficulties producing that Elixir quote directly from yecc:

• I’m in Erlang, which I know even less than Elixir,

  so for example the <expression> ::= <expression> or <bterm> production:
  I would write the corresponding action as
  {or, [context: Elixir, import: Kernel], '$1', '$3'}.
  but I get a syntax error before: 'or',
  and once I replace the atom with the string "or", just to see what other problems there are, I get two illegal expression.

  This one works, but is obviously not what I need:
  {"or", [context, "Elixir", import, "Kernel"], '$1', '$3'}.

• I miss the leading ‘c.’ (and would not want to ask the user to add it).

  I guess than a function in the Erlang code. section can solve this one.

Some quick comments:

• in Erlang or is a reserved word, hence the syntax error, so to get the atom you need to write 'or'.
• the syntax [context: Eiixir, import: Kernel] is illegal so you would have to write [{context,'Elixir'},{import,'Elixir.Kernel'}] to get the corresponding structure. Erlang has very few special syntax cases like Elixir property lists.

c. ?

single quoting the or works, thank you. and yes I remembered that the [a: b] was a reduced representation of something else. I could just nor remember what.

the leading c. is the part from c in <table-name> of the Ecto query I’m reconstructing.

in Erlang I’m working with single quoted strings, and in Elixir I need double quoted ones, so I will need a conversion function. but I also need to convert single quoted strings to the corresponding atom. I will review in the light of your hint, and hope to be more specific, but it has to do with the production <query> ::= <domain> where <expression>. I have the name of the table in an Erlang single-quotes string, and I need the atom by that name. like 'City' and I need :City. see above, the third line in the quoted form of the Ecto query.

the leading domain (or c.), I can do easily, and the conversion from string to binary and string to atom I also found their names.

so I’m all set I guess,
the single quotes to produce atoms, and the syntax for associative lists, …,
I’ll report here if I manage to get anything working, or at least looking like something that could work.

thank you all!

I’m almost there, sorry for the interruptions.

iex> "city where country.code='se' or (country='USA' and name='New York')" |>
...> to_charlist() |> 
...> :lexer.string() |> 
...> (fn {_, x, _} -> x end).() |> 
...> :parser.parse() |> 
...> (fn {_, x} -> x end).()
{:from, [context: Elixir, import: :"Ecto.Query"],
 [
   {:in, [context: Elixir, import: :Kernel],
    [{:domain, [], Elixir}, "city"]},
   [

the rest looks fine to me, but what do I do with this import: :"Ecto.Query", which in my quoted target should look like import: Ecto.Query? same for the :Kernel, what’s that leading colon?

the yecc code looks like this:
query -> domain where expression : {from, [{context, 'Elixir'}, {import, 'Ecto.Query'}], [{in, [{context, 'Elixir'}, {import, 'Kernel'}], [{domain, [], 'Elixir'}, '$1']}, [{where, '$3'}]]}.