Enum.map over list of key/value pairs with a map as the value

As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this:

[
        {"2015-02-23",
        %{
          "1. open" => "94.6226",
          "2. high" => "95.0000",
          "3. low" => "94.1250",
          "4. close" => "94.3130",
          "5. volume" => "19943800"
        }},
        ...
]

and I’ve tried the following with no success:

def datestr_to_datetime(ticker_map) do
    ticker_map
    |> Enum.map(fn {k, {nk, nv}} -> {Date.from_iso8601!(k), {nk, Float.parse(nv)}} end)
    |> Enum.sort_by(fn {d, v} -> {{d.year, d.month, d.day}, v} end)
end

this fails at Enum.map, and I’m not sure why. Any help is greatly appreciated!

How do you think it would work? It try to match:

        {"2015-02-23",
        %{
          "1. open" => "94.6226",
          "2. high" => "95.0000",
          "3. low" => "94.1250",
          "4. close" => "94.3130",
          "5. volume" => "19943800"
        }}

to

{k, {nk, nv}}

But 2nd value in tuple is map() but you try to match it to 2-ary tuple.

1 Like

Ah, I see. So I would just need to define that as a map instead. Thanks! Is there a simple way to extract the value from a map without needing to specify its key? Essentially, I would want this to iterate through all the values without needing to specify "1. Open", "2. high", ... etc.

You can match only those keys you need, if at all, or just the map as is.

%{foo: foo} = %{foo: :foo, bar: :bar}

This will match.

A good general practice when writing data transformations is “align the shape of code with the shape of the data it processes”. In this case, your requirement starts with “a list of key-value pairs”, so write the corresponding code:

def convert_from_strings(data) do
  Enum.map(data, &convert_one_element/1)
end

def convert_one_element({key_string, stats_map}) do
  # TODO: return {new_key, new_stats_map}
  {key_string, stats_map}
end

Your next requirement: the incoming key should be converted from a string to a date with Date.from_iso8601!/1. convert_from_strings will stay the same for a while, since we’ve focused attention down to one element.

def convert_one_element({key_string, stats_map}) do
  {
    Date.from_iso8601!(key_string),
    stats_map
  }
end

Your next requirement: each value in stats_map should be converted with Float.parse/1. We can write that function first:

def convert_stats_map(stats_map) do
  stats_map
  |> Enum.map(fn {k, v} -> {k, convert_float(v)} end)
  |> Map.new()
end

def convert_float(string_value) do
  string_value
  |> Float.parse()
  |> elem(0)
end

and then hook it up:

def convert_one_element({key_string, stats_map}) do
  {
    Date.from_iso8601!(key_string),
    convert_stats_map(stats_map)
  }
end

Last requirement: the list should be sorted by year/month/day. This changes convert_from_strings, giving the final code:

def convert_from_strings(data) do
  data
  |> Enum.map(&convert_one_element/1)
  |> Enum.sort_by(fn {d, v} -> {{d.year, d.month, d.day}, v} end)
end

def convert_one_element({key_string, stats_map}) do
  {
    Date.from_iso8601!(key_string),
    convert_stats_map(stats_map)
  }
end

def convert_stats_map(stats_map) do
  stats_map
  |> Enum.map(fn {k, v} -> {k, convert_float(v)} end)
  |> Map.new()
end

def convert_float(string_value) do
  string_value
  |> Float.parse()
  |> elem(0)
end

Some notes:

  • to completely match the structure, there should be a convert_key_string function called from convert_one_element. All it would do is call Date.from_iso8601!, so I wrote it inline.

  • consider making most of these convert_* functions private

  • the Access protocol and the associated functions in Kernel can DRY up some of this quite a bit:

def convert_from_strings_with_access(data) do
  import Access

  data
  |> update_in([all(), elem(0)], &Date.from_iso8601!/1)
  |> update_in([all(), elem(1)], &convert_stats_map/1)
  |> Enum.sort_by(fn {d, v} -> {{d.year, d.month, d.day}, v} end)
end

Sadly there’s no equivalent of Access.all() for “every value in a Map”, or this wouldn’t need convert_stats_map even.

4 Likes

Excellent, thank you for the detailed answer!

Just to let you know:

enumerable
|> Enum.map(&fun/1)
|> Map.new()

Is less idiomatic than:

enumarable
|> Map.new(&fun/1)
2 Likes