mfrasca

Parsing custom query language and query Ecto database

I’m rewriting a query parser I have already written in Python twice, with pyparsing/sqlalchemy, and with ply/django. I am now interested in seeing it at work within Elixir. I am VERY new to Elixir, so I’m afraid I’m not yet in the right mindset.

this is the target:
https://github.com/mfrasca/luke/blob/master/src/parser.yrl

and this is the corresponding ply/Python code:
https://github.com/mfrasca/ghini/blob/master/browse/searchgrammar.py

I am not sure about a large amount of issues.

when tokenizing words, I have reserved words, too. in my ply grammar, I let the user write strings quoted or unquoted, but I think I will drop this, to make things easier. or what would you suggest?

are there guidelines / better styles to follow when speaking of Terminals and Nonterminals? I would put Terminals in ALL CAPS, but what’s the impact on the code?

to make an example, is the form ‘[’ preferable to LBRACKET ?

coming from Python, I realise I have the inclination to think I’m producing an object when parsing the query string, and in the end I would evaluate the object, which would be a query. but I guess this is not the way I should think here. I would be building a data structure, which I would then feed to one or more functions (as many as the methods of my python class), defined by pattern-match.

leaving alone when we come to Ecto, where I will need to compute unions and intersections and negations of query sets… and navigating relations between tables… and implementing aggregating functions.

just as an example, these are two legal queries:
taxon where rank.id>=17 and count(verifications)>0
accession where id in [1 5 111] and count(plants.images)>0

it would be of great help getting: code contributions and reviews, reading suggestions, related GPL software sources.

21 comments

#ecto #leex #yecc

1 3737 21

2019-06-20 15:30:36 UTC

Marked As Solved

rvirding

Creator of Erlang

Some quick comments:

in Erlang or is a reserved word, hence the syntax error, so to get the atom you need to write 'or'.
the syntax [context: Eiixir, import: Kernel] is illegal so you would have to write [{context,'Elixir'},{import,'Elixir.Kernel'}] to get the corresponding structure. Erlang has very few special syntax cases like Elixir property lists.

c. ?

Post #17

Also Liked

kip

ex_cldr Core Team

Hmmmmm I think maybe @benwilson512 meant nimble_parsec. Which would be good for this project, more expressive and easier to debug than leex and yecc.

Post #2

imartinat

You should look at Filtrex, it runs ecto queries from a map of filters.
I used it in one of my project. You could also look at Forage, Mandarin + Forage - An admin tool for phoenix, it seems to be more powerful, there is no documentation but you could ask some help to the author. He answered quickly to my questions.

Post #9

kip

ex_cldr Core Team

Less code: yes, I would agree with that (I’ve done a reasonable amount of work in leex/yecc and with nimble_parsec).

Faster: Not sure I agree. Theres some strong sub-binary optiimization in nimble_parsec too, but for sure it would require proper testing of like for like to decide. Or someone more qualified in both approaches than me,

If one has been done the learning curve of leex/yecc or their even more ancient cousins then they are certainly straight forward to use (although I do find removing shift/reduce errors/warnings less than obvious sometimes).

All said, thats why hoped to indicate that if one is getting into parsing for the first time then parse combinators are more approachable and, in my opinion, easier to debug.

Post #7

Last Post!

mfrasca

I found it.
it’s the raw representation that I’m giving, in Erlang, and they start with Elixir.
so when Elixir tells me Ecto.Query, I need Elixir.Ecto.Query.

Post #21