njonsson

njonsson

StructuredIO, an Elixir API for consuming structured input streams

StructuredIO is a Hex package. What problem does it solve? In a nutshell, it simplifies working with structured input (such as markup or binary data) that streams from another process or computer. In such situations your application needs to tolerate input arriving piecemeal rather than in complete, well-formed data elements. Input fragments may even split multibyte characters and thereby cause encoding errors.

How do you tame this incidental complexity so that your application can just focus on using data elements instead of the error-prone tedium of reassembling them from fragments? StructuredIO can help.

There are two main features of this library.

  1. It provides a stateful process with a writer function for writing binary data. (If it sounds to you like IO that is part of the Elixir standard library, you’re right.)
  2. It provides a variety of reader functions for conditional reading according to a specified data structure. If a complete data structure has not (yet) been written to the process, nothing is read.

The combination of these two features makes it easy for your application to allow data to trickle into a StructuredIO process, and to consume data elements only as each arrives in its entirety.

You’ll find detailed examples in the project readme as well as in the project API reference.

Most Liked

njonsson

njonsson

Thanks! Yes, I’m now working on adding parsers that use the building blocks of the existing API. So there will be one-liner streaming access to well-known wire formats such as JSON, XML, CSV, BER-TLV, DER, and ASN.1.

CptnKirk

CptnKirk

But more generically aren’t you interested in turning a stream of TCP bytes into a stream of structured content? Structure packing is at the core of nearly every streaming protocol.

Unfortunately, I don’t see Flow as providing a convenient API for this. Although, this would still have value as a GenStage. Or at least it would be nice if there was a way for this to assume GenStage behavior, even if it is available outside of GenStage by default.

OvermindDL1

OvermindDL1

Huh, that’s a cool design, kind of like gen_tcp with a parsing step it sounds like (and no tcp). :slight_smile:

Where Next?

Popular in Announcing Top

danschultzer
In short Plug n’ play OAuth 2.0 provider library. Just set up a resource owner schema with Ecto (your user schema), install the dependen...
New
treble37
Just looking for a little feedback on a tiny helper library I built - Sometimes I find the need to convert maps with atom keys to maps w...
New
josevalim
Yes, yet another parser combinator library! Most of the parser combinators in the ecosystem are either compile-time, often using AST tra...
159 19870 141
New
sasajuric
I’d like to announce a small library called boundaries. This is an experimental project which explores the idea of enforcing boundaries ...
New
kevinlang
Hey all, We have made an Ecto3 Adapter for SQLite3, ecto_sqlite3! We have successfully on-boarded the full suite of integration tests (...
New
nikokozak
Hello all, I’ve been working on Svonix - a library for quickly integrating Svelte components into Phoenix views. It’s a much-needed succ...
New
MRdotB
I needed to reuse React components from my Chrome extension in my Phoenix/LiveView backend. I noticed that for Svelte/Vue, there are live...
New

Other popular topics Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement