derek-zhou
Just figured out a tricky bug, which I’d like to share with you. I read binary data from a port, and I know the program on the other side is sending nothing but utf-8 strings. However, the list of binary data I received over time may not be a valid chardata!
The reason is some unicode multi-byte sequence may straddle binary packet boundary. This usually don’t happen if the writer is slow enough because in all likelihood, the writer is writing valid string one at a time. However, if the writer is fast or the reader is slow, this improper fragmentation is bound to happen.
Where Next?
Trending in Discussions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Master modern tools, write cleaner code, and tackle real-world challenges.
**Building the future of software with AI, trust, and human connection.**
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
kip
Good point! I’ve had a similar situation in the past and found I had to leverage
String.chunk/2with the:validflag to accumulate valid UTF8 while holding over invalid bytes until the next packet comes in and so on.derek-zhou
Yeah. To parse UTF-8 input one character at a time, I still have to concatenate binaries together, which is a memory copy. before I was falsely assuming that I could just do a
:string.next_codepoint/1directly and skip the memory copy.