Using leex and yecc on Elixir

Questions / Help
, , ,
1

I am trying to tokenise a string for example ‘{}’, however the output tokens are being malformed.

Having Definitions :

OPEN_CURLY_BRACKET = \{

CLOSE_CURLY_BRACKET = \}

Rules :

{OPEN_CURLY_BRACKET}    : {token, {'{',  TokenLine}}.

{CLOSE_CURLY_BRACKET}   : {token, {'}',  TokenLine}}.

Output :

iex(10)> :lexer.string('{}')     
{:ok, ["{": 1, "}": 1], 1}

Required Output :

iex(10)> :lexer.string('{}')     
{:ok, [{:"{", 1}, {:"}",1}}, 1}

Could you kindly guide me in what the error could be please ?

Thanks a lot.

2

I have tried altering the rules to :

{OPEN_CURLY_BRACKET}    : {token, {"\{",  TokenLine}}.

{CLOSE_CURLY_BRACKET}   : {token, {"\}",  TokenLine}}.

and the output result is as follows :

iex(53)> :lexer.string('{}')     
{:ok, [{'{', 1}, {'}', 1}], 1}

So we are getting to the required output :slight_smile:

3

Your required output is not valid elixir, as :"}": is not a known construct in the language, also you are missing a closing ], but have a closing } too much.

Therefore I think you wanted to say that you expect [{:"{", 1}, {:"}", 1}]. Which indeed is what you have.

[foo: :bar] is just nicer syntax for [{:foo, :bar}].

This is called a “keyword list”. A subset of proplists.

1 Like
4

I have another problem with regards to the lexer as when I try to do :

{:ok, tokens , _} = :lexer.string(File.read!('inputText.txt'),1)

the output result is :

** (FunctionClauseError) no function clause matching in :lists.sublist/2    

    The following arguments were given to :lists.sublist/2:

        # 1
        "def P () =\r\n    u?m[v] | u!m[v].done"

        # 2
        1

    (stdlib 3.10) lists.erl:353: :lists.sublist/2
    c:/Program Files/erl10.5/lib/parsetools-2.1.8/include/leexinc.hrl:38: :lexer.string/4

What could be the problem please ?

5

You are mixing binaries and charlist between Elixir and Erlang…

This is wrong for Elixir, but even if it was right, it would be wrong for Erlang :slight_smile:

6

Yes, leex requires that its input strings are Erlang list strings not Elixir binary strings, which are what File.read returns.

3 Likes