patryk.it
Hi guys!
My question is related to the file encoding topic. I’m working on a feature that allows users to upload a CSV file that might be in different charset. It’s not only about UTF-8 and I need to detect given encoding and then parse data.
In Golang, Python, Java, C++, Objective-C we have UniversalDetector/chardet libraries. Do we have something similar to the Elixir/Erlang usage? I’m stuck with this.
How are you working with the different charsets/encodings in Elixir applications? I know - that’s not the main problem to which was designed this language, but I won’t use erlports or AWS Lambda only for detecting charset. ![]()
Trending in Questions
One of my LiveView view’s has a very heavy javascript dependency so I want to avoid loading it for every page on the site, while still us...
New
Hello everyone!
I am new to coding with Elixir/Erlang. And i am looking for some utilities for fuzzing testing like AFL++.
Does anyone ...
New
I’m very new to Elixir, so sorry if this is a silly question:
I am wondering what happens to the mailbox of a process when the machine i...
New
How do I tell Hologram not to escape HTML when substituting into templates?
{Textile.render(@product.long_description)}
escapes the HTM...
New
I load tested a plain Phoenix chat app backed by Postgres with 100 million messages in it, everything on default settings, to see what br...
New
Background
When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error:
* assembling mark...
New
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New
Other Trending Topics
Edit: 2026 May 15 - This post is archived.
Mob is alive!!
Main docs: mob v0.7.11 — Documentation
A bit of explanation for the slightly c...
New
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly)
AI dream has many faces, but general...
New
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #elixirconf-us
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #elixirconf-eu
- #security
- #processes
- #api
- #forms
- #hex










Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
LostKobrakai
nimble_csv supports converting data to utf-8 before parsing / converting from utf-8 before dumping. By default it even ships
NimbleCSV.Spreadsheet, which uses utf-16 le to work with excel csv’s. But you’ll need to setup a module per encoding.patryk.it
Thanks for your reply.
You’re right - NimbleCSV supports encoding converting (via
:encodingoption), but my problem is that I don’t know the encoding of the incoming file. It may be UTF-8 or many different, like these with German characters.When I’m trying to parse a CSV with German characters I got
Meh. Do I need to port
chardetfrom the other languages? I’m stuck with this problem.LostKobrakai
I know that’s why I explicitly only quoted you asking for the parsing part.
Generally I’d suggest you trying utf-16 le. If the csv comes out of excel this will be the encoding. If you really need the dark arts of detecting encoding (especially if no byte order mark is set) then you’ll need to wait for the input of people more knowledgeable in that than myself.
I’ve by now just let people copy and paste out of excel into a textarea, which results in csv format getting pasted as well.
NobbZ
There is no way to get what you want.
Ask your clients to upload using a specified charset only.
If you see the single byte
0xC4, that could be a latin-1 encodedÄ, though in ISO-8859-5 (kyrillic) it would be as valid as with latin-1 though encoding a different character, theФ.Therefore it is impossible to detect the encoding without knowing the content in advance.
What you describe from the other languages, is usually a very dumb heuristic.
patryk.it
Anyway thanks a lot for trying to help! Yes, I don’t have BOM set in these files.
I’m waiting for encoding hero.
harmon25
Tried out a package today called ExMagic that is a nif wrapper around ‘libmagic’… If running the ‘file’ Unix command helps determine the text encoding (not sure if it does) you might get something meaningful from ExMagic by passing it some raw binary… could get away with just the first 2kb or so of the file
ondrej-tucek
I think that the
filecmd does, see lesmana’s reply. In shortThere are also another unix cmds: enca or uchardet. So connect them via ports…
harmon25
Nice!
Not sure if the ExMagic package is running the code with the equivalent option to
-ito actually get back the encoding - think it just returns the mime… might be an easy tweak though.evadne
The libMagic route is definitely worth considering. I have a library for that. Low profile, but working.
https://github.com/evadne/gen_magic
harmon25
I ended up using ExMagic as it is just a Niff wrapper without any processes/gen_server.
I am curious though, you mention this:
Is that really necessary? I may not fully understand Niffs - does calling a niff function get serialized somehow? I figured if you were calling the niff from say multiple beam processes they would all just invoke the C code independently of each other, and not require being wrapped in a pool for concurrent access?