patryk.it
File encoding detection library
Hi guys!
My question is related to the file encoding topic. I’m working on a feature that allows users to upload a CSV file that might be in different charset. It’s not only about UTF-8 and I need to detect given encoding and then parse data.
In Golang, Python, Java, C++, Objective-C we have UniversalDetector/chardet libraries. Do we have something similar to the Elixir/Erlang usage? I’m stuck with this.
How are you working with the different charsets/encodings in Elixir applications? I know - that’s not the main problem to which was designed this language, but I won’t use erlports or AWS Lambda only for detecting charset. 
Most Liked
NobbZ
There is no way to get what you want.
Ask your clients to upload using a specified charset only.
If you see the single byte 0xC4, that could be a latin-1 encoded Ä, though in ISO-8859-5 (kyrillic) it would be as valid as with latin-1 though encoding a different character, the Ф.
Therefore it is impossible to detect the encoding without knowing the content in advance.
What you describe from the other languages, is usually a very dumb heuristic.
- Check if the input is valid UTF-8/16/32 by corresponding validators, perhaps even helped by a BOM
- If not, fall back to a single byte encoding derived from the hosts local settings
LostKobrakai
nimble_csv supports converting data to utf-8 before parsing / converting from utf-8 before dumping. By default it even ships NimbleCSV.Spreadsheet, which uses utf-16 le to work with excel csv’s. But you’ll need to setup a module per encoding.
evadne
The libMagic route is definitely worth considering. I have a library for that. Low profile, but working.
Last Post!
harmon25
While in many cases I would agree. I think libmagic and the file(1) command in unix has been tested extremely thoroughly and if it were to crash with some arbitrary input - that would be worthy of a CVE.
If you write your own low level c implementation and do not have google security engineers fuzzing it for weeks - then yea, would probably be safer as a port…
When comparing gen_magic to ExMagic, I figured the less code wrapped around libmagic - the better.
IMO the addition of a pool, and state(gen_statem) to libmagic is unnecessary complexity - that may introduce different bugs…
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"