vegabook

vegabook

How to create an Explorer DataFrame from and Adbc Result? Is going via Adbc.Result.to_map the correct way?

I’m loving Adbc which is super fast! I’m creating Explorer DataFrames like this:

Adbc.Connection.query!(Suprabonds.AdbcConn, "Select * from trace") 
|> Adbc.Result.to_map 
|> Explorer.DataFrame.new

Just wondering if this is the best way? For large numbers of rows the to_map intermediate stage is taking quite a long time.

Does Explorer use Apache Arrow under the hood as well? If so can we go directly from an Adbc.Result to an Explorer.DataFrame without the to_map stage?

What’s the fastest way of getting from Adbc.Result to Explorer.DataFrame?

Marked As Solved

josevalim

josevalim

Creator of Elixir

Use Explorer.DataFrame.from_query!(Suprabonds.AdbcConn, "Select * from trace") instead. It will allocate the underling arrow data only once and pass it directly to Explorer.

Last Post!

aspett

aspett

When I use Explorer.DataFrame.from_query with a query that does a join across two tables that have same named columns (e.g. post.id, author.id), Explorer is unhappy because it sees two columns with the same name even though they’re from different tables.

Is there anything smart I can do to solve that? Ideally want to avoid aliasing every column.

Perhaps if the resulting DF had columns in format table.col that would help :thinking:.

e.g. SELECT p1.id, p1.title, a1.id, a1.name FROM posts p1 JOIN authors ON p1.author_id = a.id might return

#Explorer.DataFrame<
  Polars[1000 x 4]
  p1.id s64 [1, 2, 3, 4, 5, ...]
  p1.title string ["Title 1", "Title 2", "Title 3", "Title 4", "Title 5", ...]
  a1.id s64 [1, 2, 3, 4, 5, ...]
  a1.name string ["Name 1", "Name 2", "Name 3", "Name 4", "Name 5", ...]
>

Where Next?

Popular in Questions Top

joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New

Other popular topics Top

nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54921 245
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 49084 226
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement