IggyEx - official Elixir client SDK for Iggy.rs message streaming

bglusman · January 24, 2024, 2:53pm

Hey all!

Haven’t posted in a while, I got busy having kids during the pandemic and haven’t done much open source in a long time!

At work we’re considering some event streaming/event sourcing architectural changes and I wanted to get a little more experience in the area, and heard about a new up and coming platform developed in Rust for message streaming that had a really good community developing around it, so I thought might be a fun excuse to develop an elixir client for the new platform. Yesterday I merged in a kind of bare minimum shell of a rustler wrapper around the native Rust SDK/client, that can’t do anything yet but ping and login, but I thought others might be interested in contributing to it/weighing in on the architecture of a new library to provide feedback and make sure we make it convenient and idiomatic to work with in Elixir.

You can read more about iggy here https://iggy.rs/ and the repo for the prototype-in-progress client is here GitHub - iggy-rs/iggy-elixir-client: Official Elixir client SDK for Iggy.rs message streaming. (pull requests very much welcome). Anyone with rust experience is extra-welcome, as I have approximately none, but the Iggy team has been super helpful and supportive, and they have a really active discord at Iggy.rs with an #elixir-client channel

dimitarvp · January 24, 2024, 3:01pm

I read quickly about Iggy but I am not sure I understood it. Is it something like a lightweight Kafka?

bglusman · January 24, 2024, 3:06pm

There’s definitely overlap/parallels with kafka, and there was even some discussion about adding a layer of API compatibility with kafka. I’m not expert at either, but, broadly yes it’s in that general direction.

bglusman · January 24, 2024, 3:18pm

FWIW, this thread on HN and linked blogpost about the origins sparked my interest

dimitarvp · January 24, 2024, 3:31pm

Yeah, same. I read through it but didn’t do much research. Sounded like a simplified message broker which definitely interests me.

entone · January 24, 2024, 3:38pm

I’m surprised to see the binary schema doesn’t include any sort of message integrity checks, CRC or something similar.

Looks like a straightforward enough schema that it would be worth implementing it using binary pattern matching.

bglusman · January 24, 2024, 4:02pm

edited: I asked on the discord, spetz the main creator responded they have have configurable CRC checks that can be enabled or disabled, though it looks like the default is disabled, he provided these links

Also encouraged anyone interested to come discuss in discord!

github.com

iggy-rs/iggy/blob/dd2739965f27207cd06da0187bc6f3bc322e60eb/configs/server.toml#L326C1-L326C26


      
          validate_checksum = false

github.com

iggy-rs/iggy/blob/dd2739965f27207cd06da0187bc6f3bc322e60eb/server/src/streaming/segments/storage.rs#L268


      
                          message.checksum,
                          message.offset,
                      ));
                  }
                  Ok(())
              })
              .await?;
              Ok(())
          }
          
          async fn load_all_indexes(&self, segment: &Segment) -> Result<Vec<Index>, IggyError> {
              trace!("Loading indexes from file...");
              let file = file::open(&segment.index_path).await?;
              let file_size = file.metadata().await?.len() as usize;
              if file_size == 0 {
                  trace!("Index file is empty.");
                  return Ok(EMPTY_INDEXES);
              }
          
              let indexes_count = file_size / 4;
              let mut indexes = Vec::with_capacity(indexes_count);

entone · January 24, 2024, 5:14pm

oh great, thanks for looking into it!

bglusman · February 21, 2024, 6:59pm

I haven’t had much time to work on this in the past 2 weeks, but, the client is about 80 or 90% complete as far as an “alpha” goes, though being built as a thin wrapper around the official Rusk SDK, there’s not much difference in practice from a alpha/v1 and a final release candidate here I suppose. login, creating topics, streams and sending messages (in a working PR) all work, it basically just needs to poll messages and figure out a story for passing in config instead of using the SDK default config… that said, I’d love to get someone else from the community with some combination of a) more rust experience, and/or b) more time to help out/help get it over the line… once its done, the Iggy team will help take over deployment/managing the hex package, etc, but need to get it the last mile or two first. If anyone is interested, respond here or DM, or come say hi in the discord channel.

dimitarvp · February 21, 2024, 8:39pm

I took a look at the code lately and I’m curious why are the threads that work with Iggy pinned at the count of 4?

Also do you have something like a list of issues tagged with e.g. needs-help or some such?

bglusman · February 23, 2024, 4:49pm

Huh, I don’t know anything about the threads pinned to 4, guess that’s somewhere inside the Iggy rust SDK we’re using? could ask in the discord, do you have a link to where in code you’re seeing that?

I haven’t made or used issues at all, I guess I could make isssues for 2 above known needs if you think helpful

dimitarvp · February 23, 2024, 5:02pm

It’s here: iggy-elixir-client/native/iggy_nif/src/client.rs at 8e4fbb6dc718cab398887c40f23eb0c2852fa8f8 · iggy-rs/iggy-elixir-client · GitHub

And it refers to the amount of threads given to the tokio runtime, it seems. I found the limit of 4 arbitrary, hence the question.

bglusman · February 23, 2024, 7:05pm

Oh gotcha, forgot about that, I think one of the Iggy guys wrote that part just to get something working, I imagine we might make that part of the client config long term, and perhaps default to System.schedulers_online or something? but dunno, just a gut reaction, don’t really know if that’s a good default or not