derek-zhou
Fun with unicode chardata
Just figured out a tricky bug, which I’d like to share with you. I read binary data from a port, and I know the program on the other side is sending nothing but utf-8 strings. However, the list of binary data I received over time may not be a valid chardata!
The reason is some unicode multi-byte sequence may straddle binary packet boundary. This usually don’t happen if the writer is slow enough because in all likelihood, the writer is writing valid string one at a time. However, if the writer is fast or the reader is slow, this improper fragmentation is bound to happen.
Most Liked
kip
Good point! I’ve had a similar situation in the past and found I had to leverage String.chunk/2 with the :valid flag to accumulate valid UTF8 while holding over invalid bytes until the next packet comes in and so on.
derek-zhou
Yeah. To parse UTF-8 input one character at a time, I still have to concatenate binaries together, which is a memory copy. before I was falsely assuming that I could just do a :string.next_codepoint/1 directly and skip the memory copy.
Popular in Discussions
Other popular topics
Chat & Discussions>Discussions
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"