How to create an Explorer DataFrame from and Adbc Result? Is going via Adbc.Result.to_map the correct way?

Questions & Help Questions
,
1

I’m loving Adbc which is super fast! I’m creating Explorer DataFrames like this:

Adbc.Connection.query!(Suprabonds.AdbcConn, "Select * from trace") 
|> Adbc.Result.to_map 
|> Explorer.DataFrame.new

Just wondering if this is the best way? For large numbers of rows the to_map intermediate stage is taking quite a long time.

Does Explorer use Apache Arrow under the hood as well? If so can we go directly from an Adbc.Result to an Explorer.DataFrame without the to_map stage?

What’s the fastest way of getting from Adbc.Result to Explorer.DataFrame?

2

Use Explorer.DataFrame.from_query!(Suprabonds.AdbcConn, "Select * from trace") instead. It will allocate the underling arrow data only once and pass it directly to Explorer.

3

Needs an empty list for options but otherwise works great. Thanks.

Explorer.DataFrame.from_query!(Suprabonds.AdbcConn, "Select * from trace", [])
4

When I use Explorer.DataFrame.from_query with a query that does a join across two tables that have same named columns (e.g. post.id, author.id), Explorer is unhappy because it sees two columns with the same name even though they’re from different tables.

Is there anything smart I can do to solve that? Ideally want to avoid aliasing every column.

Perhaps if the resulting DF had columns in format table.col that would help :thinking:.

e.g. SELECT p1.id, p1.title, a1.id, a1.name FROM posts p1 JOIN authors ON p1.author_id = a.id might return

#Explorer.DataFrame<
  Polars[1000 x 4]
  p1.id s64 [1, 2, 3, 4, 5, ...]
  p1.title string ["Title 1", "Title 2", "Title 3", "Title 4", "Title 5", ...]
  a1.id s64 [1, 2, 3, 4, 5, ...]
  a1.name string ["Name 1", "Name 2", "Name 3", "Name 4", "Name 5", ...]
>