I’m excited to share Chord, an Elixir library I’ve been working on that simplifies real-time context management and state synchronization for distributed and real-time applications. It’s designed to be both flexible and efficient, with an API that’s easy to use and integrate.
Why Chord?
Chord was born out of the need to streamline state synchronization and lifecycle management for real-time systems, where contexts change frequently. Whether you’re building a collaborative application, a real-time dashboard, or a messaging system, Chord provides the tools to manage state seamlessly.
Key Features
Context Management: Create, update, synchronize, and delete contexts effortlessly.
Delta Tracking: Minimize data transfer by only sending changes.
Flexible Backends: Use ETS (in-memory) or Redis (distributed) for storage.
Partial Updates: Update specific fields within a context without replacing the entire structure.
Customizable Cleanup: Automatically or manually clean up expired data.
Context Export & Restore: Export contexts to external storage or restore them when needed.
Pluggable Architecture: Use custom modules for time providers, delta formatting, and external state restoration.
Open to Feedback
This is the first release of Chord, and I’d love to hear your thoughts! Is there something you’d like to see added or improved? Are there edge cases I haven’t considered? Your feedback would mean a lot and help make Chord even better.
Thanks for sharing. And looks like it would be quite straightforward to add a custom backend for CubDB. The most interesting stuff seems to be the delta management, which I guess is particularly useful for non-Ecto backends like the provided ETS and Redis.
@dimitarvp I completely agree with you. I’ll be adding more real-world examples to the documentation soon. I’m also working on an interactive tool for Chord that will demonstrate its potential in a hands-on way, making it easier to understand its use cases.
@l3nz , great question! I chose the term “context” because it’s a flexible concept that can represent a different scenarios such as a group chat, VOIP call, video game session, and more. This flexibility ensures the library can be adapted to diverse scenarios without being tied to a specific domain.
Thanks again for sharing your thoughts, I really appreciate it!
This looks very interesting! I have a few questions
What do you mean by context? Context tends to be a word that has many meanings so would be helpful to clarify what that means. Is it just a container for state?
We currently have a bespoke solution over phoenix channels where we sync GenServer state from our server over to the client using json diffs. Would chord be a potential replacement to our solution?
I think I see what you mean, but in the end if I get it right it looks a bit like an Elixir Agent? I see there are mentions of deltas and collection of unused/expired ones, but what is the advantage of using those? I’m sure there is one, I just don’t see it in the docs.
You’re absolutely right, “context” is a term that can mean many things depending on the domain. In Chord, a context is indeed a container for state. For example:
In a chat application, a context could represent a group chat, holding its metadata (e.g., participants, topic, etc.) and messages.
In a game session, a context could represent the state of the game (e.g., player positions, scores, etc.).
In a collaborative document editor, it could represent the document’s state, tracking edits and updates.
From what you describe, Chord sounds like a strong candidate for your use case. It provides:
Delta tracking: Chord tracks changes to the state (deltas) and only sends what’s changed, reducing the amount of data sent over the wire.
Flexible backends: You can choose in-memory storage (ETS) or distributed backends like Redis, depending on your scalability needs or even implement your own backend.
Real-time synchronization: You can push state updates to clients through Phoenix Channels, similar to what you’re already doing.
Stateless vs. Stateful Modes: Chord can work directly with a backend or plug into your existing GenServer-based solution, giving you flexibility depending on your architecture.
To make things easier to understand, I’ve also created a test tool called ChordLab, where you can see Chord in action. Currently, it demonstrates a chat simulation that includes public and private conversations, real-time syncing, and more. You can find the repository and instructions for running it on GitHub:
I see what you mean about it resembling an Elixir Agent, but there are a few key differences and advantages that Chord brings, especially with deltas and their management:
Flexibility in architecture: While Agents are a stateful abstraction, Chord works in both stateful (via GenServer) and stateless modes (direct calls to backends like Redis or ETS). This flexibility makes it easier to adapt Chord to a variety of use cases.
Efficient state updates with deltas: Instead of syncing the entire state on every update, Chord tracks the changes (deltas). This minimizes bandwidth usage and makes real-time synchronization more efficient, especially for larger datasets or high-frequency updates.
Delta retention and expiry: Chord allows you to configure how long deltas are retained and when expired deltas should be cleaned up. This gives you fine-grained control over memory and storage, which is something Agents don’t provide by default.
Decoupled backend support: Chord allows you to plug in different backends, such as ETS for in-memory storage or Redis for distributed setups. This makes it highly scalable, which can be challenging to achieve with simple Agents.
Let me know if there are specific points you’d like to see covered more in-depth in the docs!
Hi @l3nz ,Thanks for the feedback! You’re absolutely right . I’ve updated the README with comprehensive examples to make it easier to understand and use. Feel free to check it out and let me know if there’s anything else you’d like to see!
One thing that would IMO make Chord even better is if the delta for a removed action includes the full value as it existed before it was removed (and probably put it under the old_value key as it is for the updated action).
…Apropos, what is a “backend” in Chord, by the way?
(Also, at this point I’d think “state” is a better word than “context” but I understand that it would be too big of an endeavor to rename. And, what name is better is, for the better or worse, a highly subjective matter.)
Hi @dimitarvp, thank you so much for your thoughtful feedback, I truly appreciate it!
Initially, I chose not to include old_value for the :removed action by default, instead offering the option to implement a custom delta formatter to adapt deltas to specific needs. However, after reflecting on your suggestion, I completely agree with you. Adding old_value to the :removed action provides greater flexibility and aligns better with the behavior of other actions like :modified. I’ll be incorporating this change in the next update.
Regarding your question, a “backend” in Chord simply refers to the underlying data storage mechanism, such as ETS or Redis.
As for “context” versus “state,” I understand your perspective, and you raise a great point. The term “context” was chosen to highlight the library’s flexibility and adaptability, aligning with its design goals. While “state” might feel more universally intuitive, “context” reflects the broader intent behind Chord’s functionality. I really appreciate your input, it’s always valuable to revisit and reflect on these choices.
Thanks again for your insights! Let me know if there’s anything else you’d like to discuss or suggest, I’d be happy to hear it!
