sbaildon
Partitioning data frames - is there a way to efficiently split a dataframe into multiple?
Elixir friends — I have a dataframe where I need to send different partitions based on a grouped column to different systems. Is there a way to efficiently split a dataframe into multiple?
Polars has partition_by and R has group_split but neither appear to be available in Explorer. Do I have other options?
DataFrame.from_query!(conn, “select * from events;”)
|> DataFrame.group_by(:event_type)
|> ...split into multiple frames
|> Enum.each(&dispatch_to_subsystem/1)
Marked As Solved
billylanchantin
We do appear to be missing this functionality!
Here’s an example workaround:
require Explorer.DataFrame, as: DF
require Explorer.Series, as: S
df = Explorer.Datasets.iris()
column = “species”
for value <- df[column] |> S.distinct |> S.to_list, into: %{} do
{value, df |> DF.filter(col(^column) == ^value) |> DF.discard(column)}
end
Also Liked
sbaildon
Last Post!
sbaildon
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"