chris-menz
Error decoding data from external source
Hi all, I’m trying to grab some data from a txt file online and running into a strange error when trying to use Jason.decode and Poison.decode. Here is code to reproduce and then the error.
{:ok, %HTTPoison.Response{status_code: 200, body: body}} = HTTPoison.get("https://www.ndbc.noaa.gov/data/realtime2/44097.spec")
body |> String.split("\n") |> Enum.map(fn row -> String.split(row, " ", trim: true) end) |> Enum.drop_every(2_500) |> Enum.drop_every(2_500) |> Jason.decode!()
(Jason.DecodeError) unexpected byte at position 16: 0x2E (".")
Marked As Solved
dimitarvp
Well, the data is not JSON. Even after you split it in pieces, those pieces are still not valid JSON.
Working with the first non-commented out text row:
iex(1)> text = "2023 07 27 15 56 1.1 0.1 10.5 1.1 4.8 S SW VERY_STEEP 4.1 217"
"2023 07 27 15 56 1.1 0.1 10.5 1.1 4.8 S SW VERY_STEEP 4.1 217"
iex(2)> String.split(text, " ", trim: true)
["2023", "07", "27", "15", "56", "1.1", "0.1", "10.5", "1.1", "4.8", "S", "SW",
"VERY_STEEP", "4.1", "217"]
Decoding JSON requires the input to be either JSON array or JSON object. Example:
iex(3)> "[\"2023\", \"07\"]" |> Jason.decode!()
["2023", "07"]
So your error is very logical. What’s your expected output per row? This looks more like a case of a CSV data (with space or tab separator) and not JSON.
Also Liked
dimitarvp
To be fair to all sides, it’s not necessary to convert a list record to a map at all. You can just feed each record to a function that accepts it in its original list form and then deconstruct it inside:
def process_record(record) when is_list(record) do
[year, month, day, hour, minute, ...] = record
end
And then work with each variable.
Anyhow, this is of interest to me so I will probably post sample code later.
Last Post!
codeanpeace
Word list sigils paired with Enum.zip or Enum.zip_with could be nice here.
[_columns, _units | spectral_data] =
body
|> String.split("\n", trim: true)
|> Enum.map(&String.split(&1, " ", trim: true))
...
# word list sigil
column_headers = ~w[
year month day hour minute
wave_height swell_height swell_period
wind_wave_height wind_wave_period
swell_direction wind_wave_direction
steepness average_period mean_wave_direction
]a # type modifier for atoms
first_row_map =
column_headers
|> Enum.zip(hd(spectral_data))
|> Enum.into(%{})
# or
first_row_map =
spectral_data
|> hd() # or `Enum.at(0)`
|> then(fn first_row -> Enum.zip(column_headers, first_row) end)
|> Enum.into(%{})
# or even
first_row_map =
spectral_data
|> hd() # or `Enum.at(0)`
|> Enum.zip_with(column_headers, fn [c, h] -> {h, c} end)
|> Enum.into(%{})
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex









