Haven’t posted in a while, I got busy having kids during the pandemic and haven’t done much open source in a long time!
At work we’re considering some event streaming/event sourcing architectural changes and I wanted to get a little more experience in the area, and heard about a new up and coming platform developed in Rust for message streaming that had a really good community developing around it, so I thought might be a fun excuse to develop an elixir client for the new platform. Yesterday I merged in a kind of bare minimum shell of a rustler wrapper around the native Rust SDK/client, that can’t do anything yet but ping and login, but I thought others might be interested in contributing to it/weighing in on the architecture of a new library to provide feedback and make sure we make it convenient and idiomatic to work with in Elixir.
There’s definitely overlap/parallels with kafka, and there was even some discussion about adding a layer of API compatibility with kafka. I’m not expert at either, but, broadly yes it’s in that general direction.
edited: I asked on the discord, spetz the main creator responded they have have configurable CRC checks that can be enabled or disabled, though it looks like the default is disabled, he provided these links
I haven’t had much time to work on this in the past 2 weeks, but, the client is about 80 or 90% complete as far as an “alpha” goes, though being built as a thin wrapper around the official Rusk SDK, there’s not much difference in practice from a alpha/v1 and a final release candidate here I suppose. login, creating topics, streams and sending messages (in a working PR) all work, it basically just needs to poll messages and figure out a story for passing in config instead of using the SDK default config… that said, I’d love to get someone else from the community with some combination of a) more rust experience, and/or b) more time to help out/help get it over the line… once its done, the Iggy team will help take over deployment/managing the hex package, etc, but need to get it the last mile or two first. If anyone is interested, respond here or DM, or come say hi in the discord channel.
Oh gotcha, forgot about that, I think one of the Iggy guys wrote that part just to get something working, I imagine we might make that part of the client config long term, and perhaps default to System.schedulers_online or something? but dunno, just a gut reaction, don’t really know if that’s a good default or not