njonsson

njonsson

StructuredIO, an Elixir API for consuming structured input streams

StructuredIO is a Hex package. What problem does it solve? In a nutshell, it simplifies working with structured input (such as markup or binary data) that streams from another process or computer. In such situations your application needs to tolerate input arriving piecemeal rather than in complete, well-formed data elements. Input fragments may even split multibyte characters and thereby cause encoding errors.

How do you tame this incidental complexity so that your application can just focus on using data elements instead of the error-prone tedium of reassembling them from fragments? StructuredIO can help.

There are two main features of this library.

  1. It provides a stateful process with a writer function for writing binary data. (If it sounds to you like IO that is part of the Elixir standard library, you’re right.)
  2. It provides a variety of reader functions for conditional reading according to a specified data structure. If a complete data structure has not (yet) been written to the process, nothing is read.

The combination of these two features makes it easy for your application to allow data to trickle into a StructuredIO process, and to consume data elements only as each arrives in its entirety.

You’ll find detailed examples in the project readme as well as in the project API reference.

Most Liked

njonsson

njonsson

Thanks! Yes, I’m now working on adding parsers that use the building blocks of the existing API. So there will be one-liner streaming access to well-known wire formats such as JSON, XML, CSV, BER-TLV, DER, and ASN.1.

CptnKirk

CptnKirk

But more generically aren’t you interested in turning a stream of TCP bytes into a stream of structured content? Structure packing is at the core of nearly every streaming protocol.

Unfortunately, I don’t see Flow as providing a convenient API for this. Although, this would still have value as a GenStage. Or at least it would be nice if there was a way for this to assume GenStage behavior, even if it is available outside of GenStage by default.

OvermindDL1

OvermindDL1

Huh, that’s a cool design, kind of like gen_tcp with a parsing step it sounds like (and no tcp). :slight_smile:

Where Next?

Popular in Announcing Top

josevalim
Hi everyone, We would like to announce that Plataformatec is working on a new MySQL driver called MyXQL. Our goal is to eventually integ...
New
dbern
I’m excited to announce that TaxJar has developed and open-sourced DateTimeParser. We developed it because we found a need to parse user ...
New
mplatts
With HEEX released we decided to start a components library using Tailwind CSS - check it out here: Petal Components. We also have a boi...
New
mikehostetler
I’m excited to announce Jido, a framework providing foundational primitives for building autonomous agent systems in Elixir. While develo...
New
Qqwy
Today I realized that it would be possible to implement currying-capability in Elixir, using some clever anonymous function creation. (‘c...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52341 488
New
Qqwy
Hello everyone, I wrote a small library today called MapDiff. It returns a map listing the (smallest amount of) changes to get from map...
New
Flo0807
Hello everyone! I am excited to share our heart project Backpex with you. After building several Phoenix applications, we realized that...
New
OvermindDL1
Been making an MLElixir thing (not released yet…) for fun in spare time in the past day. I’m just trying to see how much I can get an ML...
132 13966 106
New
kevinlang
Hey all, We have made an Ecto3 Adapter for SQLite3, ecto_sqlite3! We have successfully on-boarded the full suite of integration tests (...
New

Other popular topics Top

TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41539 114
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36128 110
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 126479 1222
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New

We're in Beta

About us Mission Statement