njonsson

njonsson

StructuredIO, an Elixir API for consuming structured input streams

StructuredIO is a Hex package. What problem does it solve? In a nutshell, it simplifies working with structured input (such as markup or binary data) that streams from another process or computer. In such situations your application needs to tolerate input arriving piecemeal rather than in complete, well-formed data elements. Input fragments may even split multibyte characters and thereby cause encoding errors.

How do you tame this incidental complexity so that your application can just focus on using data elements instead of the error-prone tedium of reassembling them from fragments? StructuredIO can help.

There are two main features of this library.

  1. It provides a stateful process with a writer function for writing binary data. (If it sounds to you like IO that is part of the Elixir standard library, you’re right.)
  2. It provides a variety of reader functions for conditional reading according to a specified data structure. If a complete data structure has not (yet) been written to the process, nothing is read.

The combination of these two features makes it easy for your application to allow data to trickle into a StructuredIO process, and to consume data elements only as each arrives in its entirety.

You’ll find detailed examples in the project readme as well as in the project API reference.

Most Liked

njonsson

njonsson

Thanks! Yes, I’m now working on adding parsers that use the building blocks of the existing API. So there will be one-liner streaming access to well-known wire formats such as JSON, XML, CSV, BER-TLV, DER, and ASN.1.

CptnKirk

CptnKirk

But more generically aren’t you interested in turning a stream of TCP bytes into a stream of structured content? Structure packing is at the core of nearly every streaming protocol.

Unfortunately, I don’t see Flow as providing a convenient API for this. Although, this would still have value as a GenStage. Or at least it would be nice if there was a way for this to assume GenStage behavior, even if it is available outside of GenStage by default.

OvermindDL1

OvermindDL1

Huh, that’s a cool design, kind of like gen_tcp with a parsing step it sounds like (and no tcp). :slight_smile:

Where Next?

Popular in Announcing Top

mathieuprog
Hello :wave: Allow me to introduce you to Tz, an alternative time zone database support to Tzdata. Why another library? First and fore...
New
devonestes
Introducing assertions, the library that helps you write really great test assertions! GitHub: https://github.com/devonestes/assertions ...
New
riverrun
I’ve just released version 3 of Comeonin, a password hashing library. The following small changes have been made: changes to the NIF c...
New
bryanjos
Hi, I just published version 0.23.0 of Elixirscript. Most of the changes are around JavaScript interop now that Elixirscript uses the ...
New
oltarasenko
Dear Elixir community, After a year of development, bug fixes, and improvements, we are proudly ready to share the release of Crawly 0.1...
New
Hal9000
Here is my first stab at this. README pasted below. https://github.com/Hal9000/elixir_random Comments and critiques are welcome. Th...
New
zoltanszogyenyi
Hey everyone :wave: Excited to join this forum - I am one of the founders and current project maintainers of a popular and open-source U...
New
handnot2
Samly can be used to enable SAML 2.0 Single Sign On in a Plug/Phoenix application. This library uses Erlang esaml to provide plug enabl...
New
kevinlang
Hey all, We have made an Ecto3 Adapter for SQLite3, ecto_sqlite3! We have successfully on-boarded the full suite of integration tests (...
New
wfgilman
I’ve cleaned up and open sourced three financial libraries I was using for my company. They are bindings for the APIs of these three comp...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
hariharasudhan94
Lets say i have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => "XX...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement