vegabook

vegabook

How to create an Explorer DataFrame from and Adbc Result? Is going via Adbc.Result.to_map the correct way?

I’m loving Adbc which is super fast! I’m creating Explorer DataFrames like this:

Adbc.Connection.query!(Suprabonds.AdbcConn, "Select * from trace") 
|> Adbc.Result.to_map 
|> Explorer.DataFrame.new

Just wondering if this is the best way? For large numbers of rows the to_map intermediate stage is taking quite a long time.

Does Explorer use Apache Arrow under the hood as well? If so can we go directly from an Adbc.Result to an Explorer.DataFrame without the to_map stage?

What’s the fastest way of getting from Adbc.Result to Explorer.DataFrame?

Marked As Solved

josevalim

josevalim

Creator of Elixir

Use Explorer.DataFrame.from_query!(Suprabonds.AdbcConn, "Select * from trace") instead. It will allocate the underling arrow data only once and pass it directly to Explorer.

Last Post!

aspett

aspett

When I use Explorer.DataFrame.from_query with a query that does a join across two tables that have same named columns (e.g. post.id, author.id), Explorer is unhappy because it sees two columns with the same name even though they’re from different tables.

Is there anything smart I can do to solve that? Ideally want to avoid aliasing every column.

Perhaps if the resulting DF had columns in format table.col that would help :thinking:.

e.g. SELECT p1.id, p1.title, a1.id, a1.name FROM posts p1 JOIN authors ON p1.author_id = a.id might return

#Explorer.DataFrame<
  Polars[1000 x 4]
  p1.id s64 [1, 2, 3, 4, 5, ...]
  p1.title string ["Title 1", "Title 2", "Title 3", "Title 4", "Title 5", ...]
  a1.id s64 [1, 2, 3, 4, 5, ...]
  a1.name string ["Name 1", "Name 2", "Name 3", "Name 4", "Name 5", ...]
>

Where Next?

Popular in Questions Top

vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

Other popular topics Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 54006 488
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New

We're in Beta

About us Mission Statement