chris-menz

chris-menz

Error decoding data from external source

Hi all, I’m trying to grab some data from a txt file online and running into a strange error when trying to use Jason.decode and Poison.decode. Here is code to reproduce and then the error.

{:ok, %HTTPoison.Response{status_code: 200, body: body}} = HTTPoison.get("https://www.ndbc.noaa.gov/data/realtime2/44097.spec")

body |> String.split("\n") |> Enum.map(fn row -> String.split(row, " ", trim: true) end) |> Enum.drop_every(2_500) |> Enum.drop_every(2_500) |> Jason.decode!()
(Jason.DecodeError) unexpected byte at position 16: 0x2E (".")

Marked As Solved

dimitarvp

dimitarvp

Well, the data is not JSON. Even after you split it in pieces, those pieces are still not valid JSON.

Working with the first non-commented out text row:

iex(1)> text = "2023 07 27 15 56  1.1  0.1 10.5  1.1  4.8   S  SW VERY_STEEP  4.1 217"
"2023 07 27 15 56  1.1  0.1 10.5  1.1  4.8   S  SW VERY_STEEP  4.1 217"
iex(2)> String.split(text, " ", trim: true)
["2023", "07", "27", "15", "56", "1.1", "0.1", "10.5", "1.1", "4.8", "S", "SW",
 "VERY_STEEP", "4.1", "217"]

Decoding JSON requires the input to be either JSON array or JSON object. Example:

iex(3)> "[\"2023\", \"07\"]" |> Jason.decode!()
["2023", "07"]

So your error is very logical. What’s your expected output per row? This looks more like a case of a CSV data (with space or tab separator) and not JSON.

Also Liked

dimitarvp

dimitarvp

To be fair to all sides, it’s not necessary to convert a list record to a map at all. You can just feed each record to a function that accepts it in its original list form and then deconstruct it inside:

def process_record(record) when is_list(record) do
  [year, month, day, hour, minute, ...] = record
end

And then work with each variable.

Anyhow, this is of interest to me so I will probably post sample code later.

Last Post!

codeanpeace

codeanpeace

Word list sigils paired with Enum.zip or Enum.zip_with could be nice here.

[_columns, _units | spectral_data] =
  body
  |> String.split("\n", trim: true)
  |> Enum.map(&String.split(&1, " ", trim: true))
  ...

# word list sigil
column_headers = ~w[
  year month day hour minute
  wave_height swell_height swell_period
  wind_wave_height wind_wave_period
  swell_direction wind_wave_direction
  steepness average_period mean_wave_direction
]a # type modifier for atoms

first_row_map =
  column_headers
  |> Enum.zip(hd(spectral_data))
  |> Enum.into(%{})
# or
first_row_map =
  spectral_data
  |> hd() # or `Enum.at(0)`
  |> then(fn first_row -> Enum.zip(column_headers, first_row) end)
  |> Enum.into(%{})
# or even
first_row_map =
  spectral_data
  |> hd() # or `Enum.at(0)`
  |> Enum.zip_with(column_headers, fn [c, h] -> {h, c} end)
  |> Enum.into(%{})

Where Next?

Popular in Questions Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Other popular topics Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36689 110
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New

We're in Beta

About us Mission Statement