docjazither

docjazither

No function clause matching in CSV.Decoding.Preprocessing.Lines.starts_sequence?/5

Hey OGs,
I have a function that loads csv files into my app. When my user used special characters, it failed
they used name like: Zoë
Here is my function:

defp read(csv) do
    csv.path
    |> Path.expand()
    |> File.stream!()
    |> CSV.decode(headers: true)
    |> Stream.map(fn {:ok, data} -> data end)
    |> Stream.reject(&is_nil/1)
    |> break_point()
    |> Enum.to_list() # It fails here
    |> Enum.uniq()
  end

Error:

The following arguments were given to CSV.Decoding.Preprocessing.Lines.starts_sequence?/5:
    
        # 1
        <<145, 32, 72, 111, 97, 100, 44, 105, 110, 115, 116, 97, 103, 114, 97, 109>>
    
        # 2
        "o"
    
        # 3
        false
    
        # 4
        44
    
        # 5
        ""
Attempted function clauses (showing 5 out of 5):
    
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, last_token, false, separator, _) when last_token == <<separator::utf8()>>
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, "", false, separator, _)
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, _, quoted, separator, sequence_start)
        defp starts_sequence?(<<head::utf8(), tail::binary()>>, _, quoted, separator, sequence_start)
        defp starts_sequence?("", _, quoted, _, sequence_start)

Apparently, it couldn’t recognise this character ë
I wished I could like make a callback for starts_sequence? to return a meaningful error or something like that..
How can I prevent this from happening ?

Marked As Solved

al2o3cr

al2o3cr

Make sure you have the latest version of csv - version 2.5.0 contains a fix for this error:

https://github.com/beatrichartz/csv/pull/107

Also Liked

al2o3cr

al2o3cr

The offending byte is the first one of the binary in the first argument of that error - 145. The preceding character was apparently o (the second argument to starts_sequence?) so a byte with value 145 is not valid UTF8.

145 happens to be a left single-quote () in Windows-1252, which is probably what the creator of the CSV intended.

There’s no good way to handle this error apart from upgrading, as the code that was added back in 2.5.0 is the error handling code!

al2o3cr

al2o3cr

AFAIK the only currently available way to deal with malformed UTF8 in CSV.decode is to pass the undocumented replacement option - it’s passed down unmodified to CSV.Decoding.Decoder where it is documented.

Prior to 2.5.0, this wasn’t an option because replacement happens in the lexer way after the starts_sequence? code in the preprocessor, so bad characters never made it to where they could be replaced.

A possible extension to CSV would be to also accept a 1-argument function as replacement, and provide it with the bad character.

Last Post!

jarvism

jarvism

Thank you for this code!!!
As a newb, seeing an actual implementation of reading the CSV is priceless!!!
The answer to this question is also very helpful!
I’m just not there yet. :smiley:

Where Next?

Popular in Questions Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

Other popular topics Top

Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement