I really missed the Ruby’s String#unpack.
Aetherus
Hermanverschooten
hauleth
ca1989
Aetherus
Something like some_string.unpack('C10Q>*').
Erlang/Elixir’s pattern matching is great, but sometimes I do want the quantifiers, like the 10 and * above.
0
hauleth
<<prefix::binary-10, rest::binary>> = some_string
[prefix | read_int64(rest)]
def read_int64(<<>>), do: []
def read_int64(<<int::native-64, rest::binary>>), do: [int | read_int64(rest)]
You 100% have option to define that 10. There is more problem with *, but it can be handled. You can also quite easily implement such unpack function on your own
1
LostKobrakai
Can also use the long form of that, which might be more expressive and can be used with variables:
a = 10
<<prefix::binary-size(a), rest::binary>> = "some_string"
0
bortzmeyer
Ridtt
Hello friends! I’m about to drop to community, under MIT licence, my library for bidirectional parse/encode communication.
Idea is you declaring shape of your binary data and set of constraints, highly dynamic ones, like dynamic length, item size, items count, dynamic variant dispatching, optionality of fields using callbacks. Callbacks may receive on request fields from current parse routine along with context options from total parse tree.
Most of the time you will work only with type conversion called ‘managed’. Most human value is possible, like number, utf8 string and so on. Does not matter actual data is utf16 and actual numbers are hidden under mask, I have abstractions to keep it complexity managed away from developer.
There is such features as receiving requested type conversion (binary version, for example) and option scoped from particular context interface.
In additional there is feature of virtual field. Virtual fields allow you declare your data as most readable and sane way independent of actual shape of binary.
Everything compiled down to elixir binary pattern matching, using graph topology I’m able to optimise most of blocks with known shape to single pattern and provide requested intermediate values only once, close as possible to usage and passing around only as function arguments, not creating any allocations or intermediate objects.
The most benefits you will get if you working with complex legacy protocol and you need flow like receive → modify → send.
It’s running in our production for few apps for half of year, and currently I’m finishing basic documentation.
Let me know if you guys are interested, I will try to finish basic docs and publish it this week!
5
Ridtt
I have just published it. It’s called BinStruct. I included example for adaptation of png parser discussed in this thread in examples folder. I will create separate thread where I will try explain questions if you guys got any.
BinStruct on hex: bin_struct | Hex
Foreword for png parsing way discussed in this topic compared to BinStruct ones:
While binary pattern matching itself is a powerful tool (and BinStruct abuses it) you can’t simply declare the job done once you’ve written a matching pattern. The problem is, you still can’t actually read it. People can’t read binary data.
Next, you’ll have to manually implement decoding, so you can finally understand the data. But even then, you still can’t modify it. So, you write code to handle modifications by hand—but now you can’t produce the binary output again. Once you’ve written your own binary serialization, you realize none of it is reusable.
If a product contains hundreds of such custom, hand-crafted implementations, it becomes an absolute nightmare to maintain and reuse.
This was the exact problem I set out to solve. While working alone on a complex protocol, I built a tool that empowers a single developer or a small team, make it possible to maintain complex protocols efficiently with few people.
Png example alternative: bin_struct/examples/png.exs at master · 4ait/bin_struct · GitHub
1