Przemek
Hi,
I’ve just published my tiny library:
https://github.com/pbudzik/eventlog
It’s an indexed log for events inspired by some Kafka concepts. It models a “stream” as a log and its offset index. It can append, read sequentially from a given offset. Seems to work just fine, however I’d like to improve write throughput. Simple benchmark shows ~8K events at 4 MB/s. Without indexing it’s a few times better.
Profiling kind of shows it’s send/call and I am aware that message passing is costly.
If any of you could see and play with my crap maybe we could improve it together 
regards,
PB
Where Next?
Trending in Discussions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Master modern tools, write cleaner code, and tackle real-world challenges with Manning books, videos, and courses. Start building today.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 1- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
jdumont
Hey!
I’ve just spent a bit of time looking through your library and getting to grips with how it works internally. It looks like a very similar solution to what I’ve been building myself, albeit nicely packaged up as a standalone lib. Mine’s currently all tangled up with the work-in-progress memory image app that I’ve been working on (thread about it here).
Where mine is a bit different is that I keep the index in memory with the ability to rebuild it directly from the log. It’s based on the KV store built in this series of articles, although I’ve made some changes to suit streaming events from a given offset rather than being a KV.
Not sure how that impacts performance as I’ve not properly benched it, but when I next get a chance I’ll swap my log out for yours, and perhaps package mine up nicely and get some tests/benchmarks around it.
Thanks for publishing this. It’s great to take a look at another way of solving the problem. I’ll be sure to return the favour once mines a little more presentable!