patryk.it

patryk.it

Hi guys!

My question is related to the file encoding topic. I’m working on a feature that allows users to upload a CSV file that might be in different charset. It’s not only about UTF-8 and I need to detect given encoding and then parse data.

In Golang, Python, Java, C++, Objective-C we have UniversalDetector/chardet libraries. Do we have something similar to the Elixir/Erlang usage? I’m stuck with this.

How are you working with the different charsets/encodings in Elixir applications? I know - that’s not the main problem to which was designed this language, but I won’t use erlports or AWS Lambda only for detecting charset. :frowning:

Showing Posts 1 to 10

LostKobrakai

LostKobrakai

nimble_csv supports converting data to utf-8 before parsing / converting from utf-8 before dumping. By default it even ships NimbleCSV.Spreadsheet, which uses utf-16 le to work with excel csv’s. But you’ll need to setup a module per encoding.

patryk.it

patryk.it OP

Thanks for your reply.
You’re right - NimbleCSV supports encoding converting (via :encoding option), but my problem is that I don’t know the encoding of the incoming file. It may be UTF-8 or many different, like these with German characters.

When I’m trying to parse a CSV with German characters I got

no function clause matching in CSV.Decoding.Preprocessing.Lines.starts_sequence?/5

Meh. Do I need to port chardet from the other languages? I’m stuck with this problem.

LostKobrakai

LostKobrakai

I know that’s why I explicitly only quoted you asking for the parsing part.

Generally I’d suggest you trying utf-16 le. If the csv comes out of excel this will be the encoding. If you really need the dark arts of detecting encoding (especially if no byte order mark is set) then you’ll need to wait for the input of people more knowledgeable in that than myself.

I’ve by now just let people copy and paste out of excel into a textarea, which results in csv format getting pasted as well.

NobbZ

NobbZ

There is no way to get what you want.

Ask your clients to upload using a specified charset only.

If you see the single byte 0xC4, that could be a latin-1 encoded Ä, though in ISO-8859-5 (kyrillic) it would be as valid as with latin-1 though encoding a different character, the Ф.

Therefore it is impossible to detect the encoding without knowing the content in advance.

What you describe from the other languages, is usually a very dumb heuristic.

  1. Check if the input is valid UTF-8/16/32 by corresponding validators, perhaps even helped by a BOM
  2. If not, fall back to a single byte encoding derived from the hosts local settings
patryk.it

patryk.it OP

Anyway thanks a lot for trying to help! Yes, I don’t have BOM set in these files.

I’m waiting for encoding hero. :smiley:

harmon25

harmon25

Tried out a package today called ExMagic that is a nif wrapper around ‘libmagic’… If running the ‘file’ Unix command helps determine the text encoding (not sure if it does) you might get something meaningful from ExMagic by passing it some raw binary… could get away with just the first 2kb or so of the file

ondrej-tucek

ondrej-tucek

I think that the file cmd does, see lesmana’s reply. In short

$ file -i *
umlaut-iso88591.txt: text/plain; charset=iso-8859-1
umlaut-utf16.txt:    text/plain; charset=utf-16le
umlaut-utf8.txt:     text/plain; charset=utf-

There are also another unix cmds: enca or uchardet. So connect them via ports…

harmon25

harmon25

Nice!
Not sure if the ExMagic package is running the code with the equivalent option to -i to actually get back the encoding - think it just returns the mime… might be an easy tweak though.

evadne

evadne

The libMagic route is definitely worth considering. I have a library for that. Low profile, but working.

https://github.com/evadne/gen_magic

harmon25

harmon25

I ended up using ExMagic as it is just a Niff wrapper without any processes/gen_server.

I am curious though, you mention this:

The Server should be run under a pool which provides concurrency and resiliency

Is that really necessary? I may not fully understand Niffs - does calling a niff function get serialized somehow? I figured if you were calling the niff from say multiple beam processes they would all just invoke the C code independently of each other, and not require being wrapped in a pool for concurrent access?

Where Next? Top

Trending in Questions Top

axelson
One of my LiveView view’s has a very heavy javascript dependency so I want to avoid loading it for every page on the site, while still us...
New
Hellowman
Hello everyone! I am new to coding with Elixir/Erlang. And i am looking for some utilities for fuzzing testing like AFL++. Does anyone ...
New
yarrichar
I’m very new to Elixir, so sorry if this is a silly question: I am wondering what happens to the mailbox of a process when the machine i...
New
pragdave
How do I tell Hologram not to escape HTML when substituting into templates? {Textile.render(@product.long_description)} escapes the HTM...
New
mandar1045
I load tested a plain Phoenix chat app backed by Postgres with 100 million messages in it, everything on default settings, to see what br...
New
Fl4m3Ph03n1x
Background When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error: * assembling mark...
New
rlipscombe
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New

Other Trending Topics Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
261 13183 112
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews