lukaszsamson
One thing I’ve been missing from python ecosystem was an ability to connect to a remote Apache Spark cluster from a locally running notebook and run queries and data transformations on the cluster.
I’d like to present a library I created: a native elixir client of Spark Connect protocol with Livebook integration via optional support for explorer and kino libraries.
Current status:
- Most of the protocol exposed and working (except python specific APIs)
- DataFrame and Column API including queries, joins, transformations, aggregations, reading and writing, jar upload
- Catalog API
- Structured Streaming
- Telemetry
- Tested with Spark 3.5, 4.0 and 4.1
Check out a demo notebook spark_ex/notebooks/spark_ex_demo.livemd at main · lukaszsamson/spark_ex · GitHub
Where Next?
Trending in Discussions
Other Trending Topics
Latest Livebook Threads
Latest on Elixir Forum
Sponsor Spotlight
Error tracking, logs, APM, uptime and cron monitoring in one dev-friendly platform. Find and fix problems before your users notice.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #ai
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"