bglusman

IggyEx - official Elixir client SDK for Iggy.rs message streaming

Hey all!

Haven’t posted in a while, I got busy having kids during the pandemic and haven’t done much open source in a long time!

At work we’re considering some event streaming/event sourcing architectural changes and I wanted to get a little more experience in the area, and heard about a new up and coming platform developed in Rust for message streaming that had a really good community developing around it, so I thought might be a fun excuse to develop an elixir client for the new platform. Yesterday I merged in a kind of bare minimum shell of a rustler wrapper around the native Rust SDK/client, that can’t do anything yet but ping and login, but I thought others might be interested in contributing to it/weighing in on the architecture of a new library to provide feedback and make sure we make it convenient and idiomatic to work with in Elixir.

You can read more about iggy here https://iggy.rs/ and the repo for the prototype-in-progress client is here GitHub - iggy-rs/iggy-elixir-client: Official Elixir client SDK for Iggy.rs message streaming. · GitHub (pull requests very much welcome). Anyone with rust experience is extra-welcome, as I have approximately none, but the Iggy team has been super helpful and supportive, and they have a really active discord at https://iggy.rs/discord with an #elixir-client channel

12 comments

#library #event-sourcing #events #client

4 917 12

2024-02-23 19:05:33 UTC

First 10 of 12 Posts!

dimitarvp

I read quickly about Iggy but I am not sure I understood it. Is it something like a lightweight Kafka?

Post #1

bglusman

There’s definitely overlap/parallels with kafka, and there was even some discussion about adding a layer of API compatibility with kafka. I’m not expert at either, but, broadly yes it’s in that general direction.

Post #2

bglusman

FWIW, this thread on HN and linked blogpost about the origins sparked my interest

Post #3

dimitarvp

Yeah, same. I read through it but didn’t do much research. Sounded like a simplified message broker which definitely interests me.

Post #4

entone

I’m surprised to see the binary schema doesn’t include any sort of message integrity checks, CRC or something similar.

Looks like a straightforward enough schema that it would be worth implementing it using binary pattern matching.

Post #5

bglusman

edited: I asked on the discord, spetz the main creator responded they have have configurable CRC checks that can be enabled or disabled, though it looks like the default is disabled, he provided these links

Also encouraged anyone interested to come discuss in discord!

github.com/apache/iggy

server/src/streaming/segments/storage.rs

dd2739965


      
                          message.checksum,
                          message.offset,
                      ));
                  }
                  Ok(())
              })
              .await?;
              Ok(())
          }
          
          async fn load_all_indexes(&self, segment: &Segment) -> Result<Vec<Index>, IggyError> {
              trace!("Loading indexes from file...");
              let file = file::open(&segment.index_path).await?;
              let file_size = file.metadata().await?.len() as usize;
              if file_size == 0 {
                  trace!("Index file is empty.");
                  return Ok(EMPTY_INDEXES);
              }
          
              let indexes_count = file_size / 4;
              let mut indexes = Vec::with_capacity(indexes_count);

Post #6

entone

oh great, thanks for looking into it!

Post #7

bglusman

I haven’t had much time to work on this in the past 2 weeks, but, the client is about 80 or 90% complete as far as an “alpha” goes, though being built as a thin wrapper around the official Rusk SDK, there’s not much difference in practice from a alpha/v1 and a final release candidate here I suppose. login, creating topics, streams and sending messages (in a working PR) all work, it basically just needs to poll messages and figure out a story for passing in config instead of using the SDK default config… that said, I’d love to get someone else from the community with some combination of a) more rust experience, and/or b) more time to help out/help get it over the line… once its done, the Iggy team will help take over deployment/managing the hex package, etc, but need to get it the last mile or two first. If anyone is interested, respond here or DM, or come say hi in the discord channel.

Post #8

dimitarvp

I took a look at the code lately and I’m curious why are the threads that work with Iggy pinned at the count of 4?

Also do you have something like a list of issues tagged with e.g. needs-help or some such?

Post #9

bglusman

Huh, I don’t know anything about the threads pinned to 4, guess that’s somewhere inside the Iggy rust SDK we’re using? could ask in the discord, do you have a link to where in code you’re seeing that?

I haven’t made or used issues at all, I guess I could make isssues for 2 above known needs if you think helpful

Post #10

Last Post!

bglusman

Oh gotcha, forgot about that, I think one of the Iggy guys wrote that part just to get something working, I imagine we might make that part of the client config long term, and perhaps default to System.schedulers_online or something? but dunno, just a gut reaction, don’t really know if that’s a good default or not

Post #12