docjazither

docjazither

No function clause matching in CSV.Decoding.Preprocessing.Lines.starts_sequence?/5

Hey OGs,
I have a function that loads csv files into my app. When my user used special characters, it failed
they used name like: Zoë
Here is my function:

defp read(csv) do
    csv.path
    |> Path.expand()
    |> File.stream!()
    |> CSV.decode(headers: true)
    |> Stream.map(fn {:ok, data} -> data end)
    |> Stream.reject(&is_nil/1)
    |> break_point()
    |> Enum.to_list() # It fails here
    |> Enum.uniq()
  end

Error:

The following arguments were given to CSV.Decoding.Preprocessing.Lines.starts_sequence?/5:
    
        # 1
        <<145, 32, 72, 111, 97, 100, 44, 105, 110, 115, 116, 97, 103, 114, 97, 109>>
    
        # 2
        "o"
    
        # 3
        false
    
        # 4
        44
    
        # 5
        ""
Attempted function clauses (showing 5 out of 5):
    
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, last_token, false, separator, _) when last_token == <<separator::utf8()>>
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, "", false, separator, _)
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, _, quoted, separator, sequence_start)
        defp starts_sequence?(<<head::utf8(), tail::binary()>>, _, quoted, separator, sequence_start)
        defp starts_sequence?("", _, quoted, _, sequence_start)

Apparently, it couldn’t recognise this character ë
I wished I could like make a callback for starts_sequence? to return a meaningful error or something like that..
How can I prevent this from happening ?

Marked As Solved

al2o3cr

al2o3cr

Make sure you have the latest version of csv - version 2.5.0 contains a fix for this error:

https://github.com/beatrichartz/csv/pull/107

Also Liked

al2o3cr

al2o3cr

The offending byte is the first one of the binary in the first argument of that error - 145. The preceding character was apparently o (the second argument to starts_sequence?) so a byte with value 145 is not valid UTF8.

145 happens to be a left single-quote () in Windows-1252, which is probably what the creator of the CSV intended.

There’s no good way to handle this error apart from upgrading, as the code that was added back in 2.5.0 is the error handling code!

al2o3cr

al2o3cr

AFAIK the only currently available way to deal with malformed UTF8 in CSV.decode is to pass the undocumented replacement option - it’s passed down unmodified to CSV.Decoding.Decoder where it is documented.

Prior to 2.5.0, this wasn’t an option because replacement happens in the lexer way after the starts_sequence? code in the preprocessor, so bad characters never made it to where they could be replaced.

A possible extension to CSV would be to also accept a 1-argument function as replacement, and provide it with the bad character.

Where Next?

Popular in Questions Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
JulienCorb
I am trying to implement my new.html.eex file to create new posts on my website. new.html.eex: &lt;h1&gt;Create Post&lt;/h1&gt; &lt;%= ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New

Other popular topics Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30877 112
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 53690 245
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

We're in Beta

About us Mission Statement