lukaszsamson
One thing I’ve been missing from python ecosystem was an ability to connect to a remote Apache Spark cluster from a locally running notebook and run queries and data transformations on the cluster.
I’d like to present a library I created: a native elixir client of Spark Connect protocol with Livebook integration via optional support for explorer and kino libraries.
Current status:
- Most of the protocol exposed and working (except python specific APIs)
- DataFrame and Column API including queries, joins, transformations, aggregations, reading and writing, jar upload
- Catalog API
- Structured Streaming
- Telemetry
- Tested with Spark 3.5, 4.0 and 4.1
Check out a demo notebook spark_ex/notebooks/spark_ex_demo.livemd at main · lukaszsamson/spark_ex · GitHub
Where Next?
Trending in Discussions
Other Trending Topics
Latest Livebook Threads
Latest on Elixir Forum
Sponsor Spotlight
Phoenix, LiveView, multimedia pipelines, distributed systems. 400 engineers, Elixir since 2017. Available for hire.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #elixirconf-us
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #elixirconf-eu
- #processes
- #security
- #api
- #forms
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Elixir, Erlang & BEAM conferences from Code Sync. Next: ElixirConf EU 2027
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"