lukaszsamson
One thing I’ve been missing from python ecosystem was an ability to connect to a remote Apache Spark cluster from a locally running notebook and run queries and data transformations on the cluster.
I’d like to present a library I created: a native elixir client of Spark Connect protocol with Livebook integration via optional support for explorer and kino libraries.
Current status:
- Most of the protocol exposed and working (except python specific APIs)
- DataFrame and Column API including queries, joins, transformations, aggregations, reading and writing, jar upload
- Catalog API
- Structured Streaming
- Telemetry
- Tested with Spark 3.5, 4.0 and 4.1
Check out a demo notebook spark_ex/notebooks/spark_ex_demo.livemd at main · lukaszsamson/spark_ex · GitHub
Where Next?
Trending in Discussions
Other Trending Topics
Latest Livebook Threads
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #ai
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"