derek-zhou
Just figured out a tricky bug, which I’d like to share with you. I read binary data from a port, and I know the program on the other side is sending nothing but utf-8 strings. However, the list of binary data I received over time may not be a valid chardata!
The reason is some unicode multi-byte sequence may straddle binary packet boundary. This usually don’t happen if the writer is slow enough because in all likelihood, the writer is writing valid string one at a time. However, if the writer is fast or the reader is slow, this improper fragmentation is bound to happen.
Where Next?
Trending in Discussions
Other Trending Topics
Chat & Discussions>Discussions
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #graphql
- #ai
- #genstage
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 2 of 2 Posts
kip
Good point! I’ve had a similar situation in the past and found I had to leverage
String.chunk/2with the:validflag to accumulate valid UTF8 while holding over invalid bytes until the next packet comes in and so on.derek-zhou
Yeah. To parse UTF-8 input one character at a time, I still have to concatenate binaries together, which is a memory copy. before I was falsely assuming that I could just do a
:string.next_codepoint/1directly and skip the memory copy.