How can I convert Array["some", "value"] into ("some", "value")?

I’m trying to get data from a legacy table with a composite primary key.

What I want to query is this:

SELECT
	b0.*
FROM
	"users" AS b0
WHERE (b0. "id", b0. "uid")
IN (('some', 'keys'), ('other', 'keys'));

The best I could do for now was this, but it didn’t work.

> Repo.all(where(User, [c], fragment("(?,?)", field(c, :id), field(c, :uid)) in [["some","keys"], ["other", "keys"]]))
[debug] QUERY ERROR source="users" db=0.0ms queue=37.3ms
SELECT b0."id", b0."uid", b0."email", b0."name", b0."image", b0."permission", b0."enabled", b0."last_logged_in_at", b0."last_logged_out_at", b0."inserted_at", b0."updated_at" FROM "users" AS b0 WHERE ((b0."id",b0."uid") IN (ARRAY['some','keys'],ARRAY['other','keys'])) []
** (Postgrex.Error) ERROR 42883 (undefined_function) operator does not exist: record = text[]

    query: SELECT b0."id", b0."uid", b0."email", b0."name", b0."image", b0."permission", b0."enabled", b0."last_logged_in_at", b0."last_logged_out_at", b0."inserted_at", b0."updated_at" FROM "users" AS b0 WHERE ((b0."id",b0."uid") IN (ARRAY['some','keys'],ARRAY['other','keys']))

    hint: No operator matches the given name and argument types. You might need to add explicit type casts.
    (ecto_sql) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql) lib/ecto/adapters/sql.ex:562: Ecto.Adapters.SQL.execute/5
    (ecto) lib/ecto/repo/queryable.ex:177: Ecto.Repo.Queryable.execute/4
    (ecto) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3

How can I convert Array[“some”, “value”] into (“some”, “value”)?

Hi,
I will try to answer the other part. i.e convert Array[“some”, “value”] into (“some”, “value”)?

Lets say your intent is to send this to some_function(param_1, param_2)
so you can do this

Interactive Elixir (1.9.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> arr = [1,2,3,4]
[1, 2, 3, 4]
iex(2)> List.to_tuple arr
{1, 2, 3, 4}
iex(3)> {one, two, three, four} = List.to_tuple(arr)
{1, 2, 3, 4}
iex(4)> one
1
iex(5)> three
3
iex(6)> two
2
iex(7)> some_func(one, three)
# ^^^^^^^^^ intent ^^^^^^^

Coming to your Ecto question,
There must be a better way to get a Tuple from Ecto query itself.

I believe you need to shift more of your query into the fragment.

Fundamentally, what you want to get to is:

fragment("(?,?) IN (('some', 'keys'), ('other', 'keys'))", field(c, :id), field(c, :uid))

So, to make it dynamic, do something like

fragment("(?,?) IN (?)", field(c, :id), field(c, :uid),
    ^to_query_pairs([["some", "keys"], ["other", "keys"]]))

With

defp to_query_pairs(pairs) do
  pairs
  |> Enum.map(fn [left, right] -> "(#{left}, #{right})" end)
  |> String.join(", ")
end
1 Like

Or approach the query entirely differently:

  defp query_album({artist_id, title}) do
    from(a in Album,
      where: a.artist_id == ^artist_id,
      where: a.title == ^title
    )
  end

  defp query_another(args, query),
    do: union_all(query_album(args), ^query)

  defp query_all([head | tail]),
    do: List.foldl(tail, query_album(head), &query_another/2)

  def play do
    [
      {2, "Portrait In Jazz"},
      {3, "Live At Montreaux"},
      {1, "Kind Of Blue"}
    ]
    |> query_all()
    |> Repo.all()
  end
$ mix run ./priv/repo/playground.exs
asn1: 5.0.9, compiler: 7.4.7, connection: 1.0.4, crypto: 4.6.1, db_connection: 2.0.5, decimal: 1.6.0, ecto: 3.0.6, ecto_sql: 3.0.5, elixir: 1.9.2, hex: 0.20.1, inets: 7.1.1, jason: 1.1.2, kernel: 6.5, logger: 1.9.2, mariaex: 0.9.1, mix: 1.9.2, music_db: 0.1.0, postgrex: 0.14.1, public_key: 1.7, ssl: 9.4, stdlib: 3.10, telemetry: 0.3.0

10:35:22.541 [debug] QUERY OK source="albums" db=11.9ms decode=0.8ms queue=0.8ms
  SELECT a0."id", a0."title", a0."inserted_at", a0."updated_at", a0."artist_id" 
  FROM "albums" AS a0 
  WHERE (a0."artist_id" = $1) AND (a0."title" = $2) 
UNION ALL (
  SELECT a0."id", a0."title", a0."inserted_at", a0."updated_at", a0."artist_id" 
  FROM "albums" AS a0 
  WHERE (a0."artist_id" = $3) AND (a0."title" = $4
 ) UNION ALL (
  SELECT a0."id", a0."title", a0."inserted_at", a0."updated_at", a0."artist_id" 
  FROM "albums" AS a0 
  WHERE (a0."artist_id" = $5) AND (a0."title" = $6))
) [1, "Kind Of Blue", 3, "Live At Montreaux", 2, "Portrait In Jazz"]
[
  %MusicDB.Album{
    __meta__: #Ecto.Schema.Metadata<:loaded, "albums">,
    artist: #Ecto.Association.NotLoaded<association :artist is not loaded>,
    artist_id: 1,
    genres: #Ecto.Association.NotLoaded<association :genres is not loaded>,
    id: 1,
    inserted_at: ~N[2019-08-03 14:35:27],
    title: "Kind Of Blue",
    tracks: #Ecto.Association.NotLoaded<association :tracks is not loaded>,
    updated_at: ~N[2019-08-03 14:35:27]
  },
  %MusicDB.Album{
    __meta__: #Ecto.Schema.Metadata<:loaded, "albums">,
    artist: #Ecto.Association.NotLoaded<association :artist is not loaded>,
    artist_id: 3,
    genres: #Ecto.Association.NotLoaded<association :genres is not loaded>,
    id: 5,
    inserted_at: ~N[2019-08-03 14:35:27],
    title: "Live At Montreaux",
    tracks: #Ecto.Association.NotLoaded<association :tracks is not loaded>,
    updated_at: ~N[2019-08-03 14:35:27]
  },
  %MusicDB.Album{
    __meta__: #Ecto.Schema.Metadata<:loaded, "albums">,
    artist: #Ecto.Association.NotLoaded<association :artist is not loaded>,
    artist_id: 2,
    genres: #Ecto.Association.NotLoaded<association :genres is not loaded>,
    id: 4,
    inserted_at: ~N[2019-08-03 14:35:27],
    title: "Portrait In Jazz",
    tracks: #Ecto.Association.NotLoaded<association :tracks is not loaded>,
    updated_at: ~N[2019-08-03 14:35:27]
  }
]
$ 
1 Like

First of all, Thank you for your suggestion. It’s a very interesting approach! :smiley: I’ll remember that for later.
But, This code is for batching and the frequency of calls will be very frequent. So this method doesn’t seem to be available in this case.
Anyway, Thanks again, I learned something here.

Looks good. I’ll try it. :dancer:

I’m not sure I understand the issue you seem to think there is.

As it is, the IN comparison is expecting a list of scalars - (b0."id",b0."uid") would be a list, not a scalar.

But even if it did work, there is another problem:

Each index in the Array of values must correspond to the same index in the Array of keys.

i.e. the query will not necessarily return the records in the same order as the criteria appear in the list - you have to specifically arrange for that:

  defp query_album(index, {artist_id, title}) do
    seq = Integer.to_string(index)
    from(a in Album,
      where: a.artist_id == ^artist_id,
      where: a.title == ^title,
      select: %{seq: ^seq, id: a.id, artist_id: a.artist_id, title: a.title}
    )
  end

  defp query_another(args, {index, query}),
    do: {index + 1, union_all(query_album(index, args), ^query)}

  defp query_all([head | tail]) do
    {_, query} = List.foldl(tail, {1, query_album(0, head)}, &query_another/2)
    order_by(query, fragment("1 ASC"))
  end

  def play do
    [
      {2, "Portrait In Jazz"},
      {3, "Live At Montreaux"},
      {1, "Kind Of Blue"}
    ]
    |> query_all()
    |> Repo.all()
    |> Enum.map(&Map.drop(&1, [:seq]))
  end
$ mix run ./priv/repo/playground.exs
asn1: 5.0.9, compiler: 7.4.7, connection: 1.0.4, crypto: 4.6.1, db_connection: 2.0.5, decimal: 1.6.0, ecto: 3.0.6, ecto_sql: 3.0.5, elixir: 1.9.2, hex: 0.20.1, inets: 7.1.1, jason: 1.1.2, kernel: 6.5, logger: 1.9.2, mariaex: 0.9.1, mix: 1.9.2, music_db: 0.1.0, postgrex: 0.14.1, public_key: 1.7, ssl: 9.4, stdlib: 3.10, telemetry: 0.3.0

19:34:58.456 [debug] QUERY OK source="albums" db=0.6ms decode=0.9ms queue=0.7ms
  SELECT $1, a0."id", a0."artist_id", a0."title" 
  FROM "albums" AS a0 
  WHERE (a0."artist_id" = $2) AND (a0."title" = $3) 
UNION ALL (
  SELECT $4, a0."id", a0."artist_id", a0."title" FROM "albums" AS a0 WHERE (a0."artist_id" = $5) AND (a0."title" = $6) 
UNION ALL (
  SELECT $7, a0."id", a0."artist_id", a0."title" 
  FROM "albums" AS a0 
  WHERE (a0."artist_id" = $8) AND (a0."title" = $9)
)) 
ORDER BY 1 ASC ["2", 1, "Kind Of Blue", "1", 3, "Live At Montreaux", "0", 2, "Portrait In Jazz"]
[
  %{artist_id: 2, id: 4, title: "Portrait In Jazz"},
  %{artist_id: 3, id: 5, title: "Live At Montreaux"},
  %{artist_id: 1, id: 1, title: "Kind Of Blue"}
]

The IN comparison can be emulated in this manner:

  defp query_album({artist_id, title}) do
    from(a in Album,
      where: a.artist_id == ^artist_id and a.title == ^title
    )
  end

  defp or_another({artist_id, title}, query),
    do: or_where(query, [a], a.artist_id == ^artist_id and a.title == ^title)

  defp query_all([head | tail]),
    do: List.foldl(tail, query_album(head), &or_another/2)

  def play do
    [
      {2, "Portrait In Jazz"},
      {3, "Live At Montreaux"},
      {1, "Kind Of Blue"}
    ]
    |> query_all()
    |> Repo.all()
  end
$ mix run ./priv/repo/playground.exs
asn1: 5.0.9, compiler: 7.4.7, connection: 1.0.4, crypto: 4.6.1, db_connection: 2.0.5, decimal: 1.6.0, ecto: 3.0.6, ecto_sql: 3.0.5, elixir: 1.9.2, hex: 0.20.1, inets: 7.1.1, jason: 1.1.2, kernel: 6.5, logger: 1.9.2, mariaex: 0.9.1, mix: 1.9.2, music_db: 0.1.0, postgrex: 0.14.1, public_key: 1.7, ssl: 9.4, stdlib: 3.10, telemetry: 0.3.0

20:59:53.957 [debug] QUERY OK source="albums" db=5.3ms decode=0.8ms queue=0.7ms
SELECT a0."id", a0."title", a0."inserted_at", a0."updated_at", a0."artist_id" 
FROM "albums" AS a0 
WHERE (((a0."artist_id" = $1) AND (a0."title" = $2))) 
  OR ((a0."artist_id" = $3) AND (a0."title" = $4)) 
  OR ((a0."artist_id" = $5) AND (a0."title" = $6)) [2, "Portrait In Jazz", 3, "Live At Montreaux", 1, "Kind Of Blue"]
[
  %MusicDB.Album{
    __meta__: #Ecto.Schema.Metadata<:loaded, "albums">,
    artist: #Ecto.Association.NotLoaded<association :artist is not loaded>,
    artist_id: 1,
    genres: #Ecto.Association.NotLoaded<association :genres is not loaded>,
    id: 1,
    inserted_at: ~N[2019-08-03 14:35:27],
    title: "Kind Of Blue",
    tracks: #Ecto.Association.NotLoaded<association :tracks is not loaded>,
    updated_at: ~N[2019-08-03 14:35:27]
  },
  %MusicDB.Album{
    __meta__: #Ecto.Schema.Metadata<:loaded, "albums">,
    artist: #Ecto.Association.NotLoaded<association :artist is not loaded>,
    artist_id: 2,
    genres: #Ecto.Association.NotLoaded<association :genres is not loaded>,
    id: 4,
    inserted_at: ~N[2019-08-03 14:35:27],
    title: "Portrait In Jazz",
    tracks: #Ecto.Association.NotLoaded<association :tracks is not loaded>,
    updated_at: ~N[2019-08-03 14:35:27]
  },
  %MusicDB.Album{
    __meta__: #Ecto.Schema.Metadata<:loaded, "albums">,
    artist: #Ecto.Association.NotLoaded<association :artist is not loaded>,
    artist_id: 3,
    genres: #Ecto.Association.NotLoaded<association :genres is not loaded>,
    id: 5,
    inserted_at: ~N[2019-08-03 14:35:27],
    title: "Live At Montreaux",
    tracks: #Ecto.Association.NotLoaded<association :tracks is not loaded>,
    updated_at: ~N[2019-08-03 14:35:27]
  }
]

but the order of the records isn’t guaranteed or easily influenced.

1 Like

It does not works for me. :sweat:

Repo.all(where(User, [c], fragment("(?, ?) IN ?", field(c, :id), field(c, :uid), ^Enum.join(Enum.map(inputs, fn [left, right] -> "(#{left}, #{right})" end), ","))))
** (Postgrex.Error) ERROR 42601 (syntax_error) syntax error at or near "$1"

    query: SELECT b0."id", b0."uid" FROM "users" AS b0 WHERE ((b0."id", b0."uid") IN $1)
    (ecto_sql) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql) lib/ecto/adapters/sql.ex:562: Ecto.Adapters.SQL.execute/5
    (ecto) lib/ecto/repo/queryable.ex:177: Ecto.Repo.Queryable.execute/4
    (ecto) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
[debug] QUERY ERROR source="users" db=0.0ms queue=0.5ms
SELECT b0."id", b0."uid"FROM "users" AS b0 WHERE ((b0."id", b0."uid") IN $1) ["(1, 2),(3, 4)"]

Shouldn’t this be

fragment("(?, ?) IN (?)"

Note the brackets after the IN.

1 Like

It still returns error. :sweat:

iex(model@127.0.0.1)5> Repo.all(where(User, [c], fragment("(?, ?) IN (?)", field(c, :id), field(c, :uid), ^Enum.join(Enum.map([[1,2],[3,4]], fn [left, right] -> "(#{left}, #{right})" end), ","))))
[debug] QUERY ERROR source="users" db=2.9ms queue=14.1ms
SELECT b0."id", b0."uid" FROM "users" AS b0 WHERE ((b0."id", b0."uid") IN ($1)) ["(1, 2),(3, 4)"]
** (DBConnection.EncodeError) Postgrex expected a tuple, got "(1, 2),(3, 4)". Please make sure the value you are passing matches the definition in your table or in your query or convert the value accordingly.
    (postgrex) lib/postgrex/type_module.ex:897: Postgrex.DefaultTypes.encode_params/3
    (postgrex) lib/postgrex/query.ex:75: DBConnection.Query.Postgrex.Query.encode/3
    (db_connection) lib/db_connection.ex:1148: DBConnection.encode/5
    (db_connection) lib/db_connection.ex:1246: DBConnection.run_prepare_execute/5
    (db_connection) lib/db_connection.ex:1342: DBConnection.run/6
    (db_connection) lib/db_connection.ex:540: DBConnection.parsed_prepare_execute/5
    (db_connection) lib/db_connection.ex:533: DBConnection.prepare_execute/4
    (ecto_sql) lib/ecto/adapters/sql.ex:570: Ecto.Adapters.SQL.execute!/4

iex(model@127.0.0.1)5> Repo.all(where(User, [c], fragment("(?, ?) IN (?)", field(c, :id), field(c, :uid), ^[[1,2],[3,4]])))                                                           [debug] QUERY ERROR source="users" db=3.4ms queue=9.6ms
SELECT b0."id", b0."uid" FROM "users" AS b0 WHERE ((b0."id", b0."uid") IN ($1)) [[[1, 2], [3, 4]]]
** (DBConnection.EncodeError) Postgrex expected a tuple, got [[1, 2], [3, 4]]. Please make sure the value you are passing matches the definition in your table or in your query or convert the value accordingly.
    (postgrex) lib/postgrex/type_module.ex:897: Postgrex.DefaultTypes.encode_params/3
    (postgrex) lib/postgrex/query.ex:75: DBConnection.Query.Postgrex.Query.encode/3
    (db_connection) lib/db_connection.ex:1148: DBConnection.encode/5
    (db_connection) lib/db_connection.ex:1246: DBConnection.run_prepare_execute/5
    (db_connection) lib/db_connection.ex:1342: DBConnection.run/6
    (db_connection) lib/db_connection.ex:540: DBConnection.parsed_prepare_execute/5
    (db_connection) lib/db_connection.ex:533: DBConnection.prepare_execute/4
    (ecto_sql) lib/ecto/adapters/sql.ex:570: Ecto.Adapters.SQL.execute!/4

You could create a custom ecto type. I know ex_money_sql uses one with a composite column.

1 Like

Let me expand on my earlier point:

WHERE (b0. "id", b0. "uid") IN (('some', 'keys'), ('other', 'keys'))

In SQL terms

(b0. "id", b0. "uid")

is a list while

(('some', 'keys'), ('other', 'keys'))

is a list of lists. As per PostgreSQL documentation

The right-hand side is a parenthesized list of scalar expressions.

So as per documentation

WHERE (b0. "id", b0. "uid") IN (('some', 'keys'), ('other', 'keys'))

is not supported in SQL given that

(('some', 'keys'), ('other', 'keys'))

is a list of lists rather than a list of scalars.


Correction: And yet

$ psql music_db
psql (11.5)
Type "help" for help.

music_db=# SELECT
music_db-#   a0."id",
music_db-#   a0."title",
music_db-#   a0."inserted_at",
music_db-#   a0."updated_at",
music_db-#   a0."artist_id"
music_db-# FROM "albums" AS a0 
music_db-# WHERE
music_db-#   (a0."artist_id", a0."title") IN (
music_db(#     (2, 'Portrait In Jazz'),
music_db(#     (3, 'Live At Montreaux'),
music_db(#     (1, 'Kind Of Blue')
music_db(#   );
 id |       title       |     inserted_at     |     updated_at      | artist_id 
----+-------------------+---------------------+---------------------+-----------
  1 | Kind Of Blue      | 2019-08-03 14:35:27 | 2019-08-03 14:35:27 |         1
  4 | Portrait In Jazz  | 2019-08-03 14:35:27 | 2019-08-03 14:35:27 |         2
  5 | Live At Montreaux | 2019-08-03 14:35:27 | 2019-08-03 14:35:27 |         3
(3 rows)

music_db=# 

(a0."artist_id", a0."title") must be interpreted as some kind of (implicit) composite type.


Furthermore

So the intent behind

WHERE (b0. "id", b0. "uid") IN (('some', 'keys'), ('other', 'keys'))

is correctly in SQL expressed as

WHERE
   (b0."id" = 'some' AND b0."uid" = 'keys')
OR (b0."id" = 'other' AND b0."uid" = 'keys')

The only problem is that this query (or IN for that matter) doesn’t guarantee the order of the returned records.

Given that the desired order isn’t based on information that is found in the database it becomes necessary to inject the value to order by. That is typically accomplished this way:

((
  SELECT 0, b0."id", b0."uid" FROM "users" AS b0 WHERE b0."id" = 'some' AND b0."uid"  = 'keys'
) UNION ALL (
  SELECT 1, b0."id", b0."uid" FROM "users" AS b0 WHERE b0."id" = 'other' AND b0."uid"  = 'keys'
)) ORDER BY 1 ASC 

UNION ALL is used instead of UNION for performance reasons - UNION will try to eliminate duplicates - UNION ALL won’t.

Ecto.Query.union/2
Ecto.Query.union_all/2

2 Likes

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/elixir-ecto/wWfbbHUgtGk
According to Jose, currently there is no way to pass tuples dynamically.
We decided not to build a complex query, just merge the columns and retrieve them as strings and fix them later if the ecto was updated.

WHERE concat(b0."id", '-', b0."uid") IN ('some-keys', 'other-keys')

Thank you for your concern. :sweat_smile:

2 Likes