Postgrex error with sql library: message: "syntax error at or near \"$0\""

Hello, I am trying to do a postgrex query with the sql library where I get the first entry from a table sorted by last_visited and excluding all entries where url is in state.lock or host is in state.politeness.

{query, params} = ~SQL[from pages]
      |> ~SQL[where url not in #{state.lock}]
      |> ~SQL[where host not in #{state.politeness}]
      |> ~SQL"select url"
      |> to_sql()

    {:ok, %Postgrex.Result{rows: [[response]]}} = Postgrex.query(:postgrex, query, params)

However, when I run this I get:

** (MatchError) no match of right hand side value: {:error, %Postgrex.Error{message: nil, postgres: %{code: :syntax_error, file: "scan.l", line: "1244", message: "syntax error at or near \"$0\"", pg_code: "42601", position: "40", routine: "scanner_yyerror", severity: "ERROR", unknown: "ERROR"}, connection_id: 7202, query: "select url from pages where url not in $0 where host not in $0"}}

Does anyone know how to fix this?

Can you print the SQL it generates ?

It looks like the query it generates is hidden in the provided error message:

1 Like

Whoops!

Right so @user974881 you can’t use SQL in with parameters like that. Instead do:

not host = any?(#{state.lock})
4 Likes

:wave:

Double where might also be a problem.

3 Likes

Hey there, as I said in the other thread Pattern match postgex result - (MatchError) no match of right hand side value: %Postgrex.Result - #14 by Schultzer this I would need a bit more time to fix, it’s not on you.

I haven’t impl. double where in the sense that it would be accumulating ands. And I would say it’s an open question of how that should work.

I appreciate you’re taking the library for a ride, it gives me a clear picture of what your expectations are and where I need to improve.

Although I’m currently prioritizing a rewrite of the parser, since the first iteration has been handwritten and does not even parse DDL.

Well, I tried this:

{query, params} = ~SQL[from pages]
      |> ~SQL[where not url = any?(#{state.lock}) and not host = any?(#{state.politeness})]
      |> ~SQL"select url"
      |> to_sql()

    {:ok, %Postgrex.Result{rows: [[response]]}} = Postgrex.query(:postgrex, query, params)

so that I only have one where and also for your solution, and now I get:

** (MatchError) no match of right hand side value: {:error, %Postgrex.Error{message: nil, postgres: %{code: :undefined_parameter, file: "parse_param.c", line: "140", message: "there is no parameter $0", pg_code: "42P02", position: "44", routine: "variable_paramref_hook", severity: "ERROR", unknown: "ERROR"}, connection_id: 4546, query: "select url from pages where not url = any ($0) and not host = any ($0)"}}

If you are on main you might be able to get away with this: sql/test/string_test.exs at 35d743c7715da9336269dfd7ea8426f4d2fa7ec6 · elixir-dbvisor/sql · GitHub

But your example might be an edge case that I haven’t gotten to yet.

now I get

** (MatchError) no match of right hand side value: {:error, %Postgrex.Error{message: nil, postgres: %{code: :syntax_error, file: "scan.l", line: "1244", message: "syntax error at or near \"$0\"", pg_code: "42601", position: "40", routine: "scanner_yyerror", severity: "ERROR", unknown: "ERROR"}, connection_id: 5621, query: "select url from pages where url not in $0 and host not in $1"}}

with

{query, params} = ~SQL[from pages]
      |> ~SQL[where url not in #{state.lock} and host not in #{state.politeness}]
      |> ~SQL"select url"
      |> to_sql()

    {:ok, %Postgrex.Result{rows: [[response]]}} = Postgrex.query(:postgrex, query, params)

You added a space that shouldn’t be there after any.

Taking a step back, are you sure this is the right library for you? The sql library seems aimed at people familiar with SQL and who prefer to use that syntax instead of the ecto query syntax. It doesn’t seem like this is syntax you’re super comfortable with though and the process of using ecto would be a lot simpler for what you’re doing.

2 Likes

I’m not using any, I’m using in(which then is converted to any by sql), and in the code I used before I do not see the extra space.

And I do think it is better I use sql as the problem is more with elixir than with SQL.

Your problem is the SQL. Take the SQL you are writing and put it in Postgres, you will get an error. Make sure to use parameters and not just write out the IN manually.

You can’t use parameters with IN like that; this is a Postgres limitation not an Elixir one. This is why you should be using any and not in when using parameters.

1 Like

Well, if this test is valid in sql, in should be converted to any:

It’s in the select tho…

Oh now I see what you mean, I did

{query, params} = ~SQL[from pages]
      |> ~SQL[where not url = any?(#{state.lock}) and not host = any?(#{state.politeness})]
      |> ~SQL"select url"
      |> to_sql()

{:ok, %Postgrex.Result{rows: [[response]]}} = Postgrex.query(:postgrex, query, params)

but what sql turns it into is:

select url from pages where not url = any? ($1) and not host = any? ($1)

which is incorrect, so that is a bug in sql.
I’ll try doing it without sql and see if it works

The workaround I have found right now is doing the following query:

SELECT url FROM pages WHERE NOT url = ANY($1) AND NOT host = ANY($2) LIMIT 1

without using the sql library, but it would be nice to be able to do it with the library.

Hey there, I’ve been hard at work on the next version. Do you mind trying out main with the new adapters:

use SQL, adapter: SQL.Adapters.Postgres
~SQ[....]

I’m aiming to get a release out next week after I have updated all the docs, but as we speak the library is conformant to SQL 2016 standard, with over 900 generated tests.

The version after this will focus on improving the API. So that we might be able to accumelate where clauses as you tried in your prior attempt.

1 Like

I did

{query, params} = ~SQL[from pages]
  |> ~SQL[where not url = any?(#{state.lock}) and not host = any?(#{state.politeness})]
  |> ~SQL"select url limit 1"
  |> to_sql()

IO.inspect({query, params})

but it just hangs…
My sql query does work, however:

query = "SELECT url FROM pages WHERE NOT url = ANY($1) AND NOT host = ANY($2) LIMIT 1"
params = [state.lock, state.politeness]

Thanks, I’m believe the issue is the question mark after any. sql - IN vs ANY operator in PostgreSQL - Stack Overflow PostgreSQL: Documentation: 17: 9.24. Subquery Expressions

I look into the the hanging part.

I had also tried without, and got the same result.
But, in the sql library, should I use the question mark or not?

Now I updated it, and I get

iex(1)> DBHandler.page_to_scrape()

14:44:34.635 [error] GenServer DBHandler terminating
** (UndefinedFunctionError) function nil.token_to_string/1 is undefined
    nil.token_to_string({:select, [line: 0, column: 6, file: nil], [{:ident, [line: 0, column: 10, file: nil], [~c"url"]}]})
    (elixir 1.18.3) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
    (sql 0.1.0) lib/sql.ex:113: String.Chars.SQL.to_string/1
    (sql 0.1.0) lib/sql.ex:41: SQL.to_sql/1
    (dbhandler 0.1.0) lib/dbhandler.ex:136: DBHandler.handle_call/3
    (stdlib 4.3.1.6) gen_server.erl:1149: :gen_server.try_handle_call/4
    (stdlib 4.3.1.6) gen_server.erl:1178: :gen_server.handle_msg/6
    (stdlib 4.3.1.6) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.573.0>): :page_to_scrape
State: %{lock: [], politeness: []}
Client #PID<0.573.0> is alive

    (stdlib 4.3.1.6) gen.erl:256: :gen.do_call/4
    (elixir 1.18.3) lib/gen_server.ex:1125: GenServer.call/3
    (elixir 1.18.3) src/elixir.erl:386: :elixir.eval_external_handler/3
    (stdlib 4.3.1.6) erl_eval.erl:748: :erl_eval.do_apply/7
    (elixir 1.18.3) src/elixir.erl:364: :elixir.eval_forms/4
    (elixir 1.18.3) lib/module/parallel_checker.ex:120: Module.ParallelChecker.verify/1
    (iex 1.18.3) lib/iex/evaluator.ex:336: IEx.Evaluator.eval_and_inspect/3
    (iex 1.18.3) lib/iex/evaluator.ex:310: IEx.Evaluator.eval_and_inspect_parsed/3
Database handler started!
** (exit) exited in: GenServer.call(DBHandler, :page_to_scrape, 60000)
    ** (EXIT) an exception was raised:
        ** (UndefinedFunctionError) function nil.token_to_string/1 is undefined
            nil.token_to_string({:select, [line: 0, column: 6, file: nil], [{:ident, [line: 0, column: 10, file: nil], [~c"url"]}]})
            (elixir 1.18.3) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
            (sql 0.1.0) lib/sql.ex:113: String.Chars.SQL.to_string/1
            (sql 0.1.0) lib/sql.ex:41: SQL.to_sql/1
            (dbhandler 0.1.0) lib/dbhandler.ex:136: DBHandler.handle_call/3
            (stdlib 4.3.1.6) gen_server.erl:1149: :gen_server.try_handle_call/4
            (stdlib 4.3.1.6) gen_server.erl:1178: :gen_server.handle_msg/6
            (stdlib 4.3.1.6) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
    (elixir 1.18.3) lib/gen_server.ex:1128: GenServer.call/3
    iex:1: (file)
iex(1)>