chris-menz
Error decoding data from external source
Hi all, I’m trying to grab some data from a txt file online and running into a strange error when trying to use Jason.decode and Poison.decode. Here is code to reproduce and then the error.
{:ok, %HTTPoison.Response{status_code: 200, body: body}} = HTTPoison.get("https://www.ndbc.noaa.gov/data/realtime2/44097.spec")
body |> String.split("\n") |> Enum.map(fn row -> String.split(row, " ", trim: true) end) |> Enum.drop_every(2_500) |> Enum.drop_every(2_500) |> Jason.decode!()
(Jason.DecodeError) unexpected byte at position 16: 0x2E (".")
Marked As Solved
dimitarvp
Well, the data is not JSON. Even after you split it in pieces, those pieces are still not valid JSON.
Working with the first non-commented out text row:
iex(1)> text = "2023 07 27 15 56 1.1 0.1 10.5 1.1 4.8 S SW VERY_STEEP 4.1 217"
"2023 07 27 15 56 1.1 0.1 10.5 1.1 4.8 S SW VERY_STEEP 4.1 217"
iex(2)> String.split(text, " ", trim: true)
["2023", "07", "27", "15", "56", "1.1", "0.1", "10.5", "1.1", "4.8", "S", "SW",
"VERY_STEEP", "4.1", "217"]
Decoding JSON requires the input to be either JSON array or JSON object. Example:
iex(3)> "[\"2023\", \"07\"]" |> Jason.decode!()
["2023", "07"]
So your error is very logical. What’s your expected output per row? This looks more like a case of a CSV data (with space or tab separator) and not JSON.
Also Liked
dimitarvp
To be fair to all sides, it’s not necessary to convert a list record to a map at all. You can just feed each record to a function that accepts it in its original list form and then deconstruct it inside:
def process_record(record) when is_list(record) do
[year, month, day, hour, minute, ...] = record
end
And then work with each variable.
Anyhow, this is of interest to me so I will probably post sample code later.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Courses that'll move you from confusion to "Aha, now I get it!"