Merging (upserting) a list of maps into an existing list of maps the right way?

Questions / Help
1

Hi all! I’m a newbie who just struggled for a day and a half implementing what I thought would be a very simple problem.

The problem: Given an existing list of maps, merge an incoming list of maps… appending to the list when a map with a new composite key (action and legal_inventories) arrives and updating a string if the composite key already exists.

Example:

iex(1)> existing_actions = [%{action: "provision", legal_inventories: ["development", "production"],
   name: "Deploy SSL Certs & Provision Postfix Relay"},
 %{action: "provision-deploy", legal_inventories: ["development", "production"],
   name: "Deploy SSL Certs & Provision Postfix Relay"}]
iex(2)> new_or_updated_actions = [%{action: "deploy", legal_inventories: ["development", "production"],
   name: "Deploy Postfix Relay Users"},
 %{action: "provision-deploy", legal_inventories: ["development", "production"],
   name: "Deploy Postfix Relay Users"}]
iex(3)> Testing.apply_new_or_updated_actions(existing_actions, new_or_updated_actions)
[%{action: "provision", legal_inventories: ["development", "production"],
   name: "Deploy SSL Certs & Provision Postfix Relay"},
 %{action: "provision-deploy", legal_inventories: ["development", "production"],
   name: "Deploy SSL Certs & Provision Postfix Relay & Deploy Postfix Relay Users"},
 %{action: "deploy", legal_inventories: ["development", "production"],
   name: "Deploy Postfix Relay Users"}]

This code works:

defmodule Testing do
  def apply_new_or_updated_actions(valid_actions, new_or_updated_actions) do
    # first update valid_actions, then insert any new actions to the list.
    valid_actions = update_valid_actions(valid_actions, [], new_or_updated_actions)
    Enum.reduce(new_or_updated_actions, valid_actions, &insert_if_new/2)
  end

  defp update_valid_actions([], new_valid_actions, _new_or_updated_actions) do
    new_valid_actions
  end
  defp update_valid_actions([head_valid_action|tail_valid_actions], new_valid_actions, new_or_updated_actions) do
    update_valid_actions(tail_valid_actions, new_valid_actions ++ apply_updates_to_valid_action(new_or_updated_actions, head_valid_action), new_or_updated_actions)
  end

  defp apply_updates_to_valid_action([], valid_action) do
    [valid_action]
  end
  defp apply_updates_to_valid_action([head_new_or_updated_action|tail_new_or_updated_action], valid_action) do
    possibly_mutated_action =
      if action_match?(head_new_or_updated_action, valid_action) do
        Map.update(valid_action, :name, head_new_or_updated_action.name, &(&1 <> " & " <> head_new_or_updated_action.name))
      else
        valid_action
      end
    apply_updates_to_valid_action(tail_new_or_updated_action, possibly_mutated_action)
  end

  defp insert_if_new(new_or_updated_action, valid_actions) do
    if action_match?(new_or_updated_action, valid_actions), do: valid_actions, else: valid_actions ++ [new_or_updated_action]
  end

  defp action_match?(action, list_of_actions) when is_list(list_of_actions) do
    Enum.any?(list_of_actions, &((action.action == &1.action) and (action.legal_inventories == &1.legal_inventories)))
  end
  defp action_match?(action1, action2) when is_map(action2) do
    (action1.action == action2.action) and (action1.legal_inventories == action2.legal_inventories)
  end
end

I’ve tried many simpler solutions using functions from Enum and for list comprehensions before settling on a nested recursive pattern. This works but it seems overly complex and I feel there is a better and more idiomatic pattern. Also, I’m aware my variable and function names are too long but I needed them to be descriptive to be able to reason about the code.

I will be doing many similar data transformations like these in the future and would really appreciate if anyone can help point out better ways to do this.

Thanks!!!

1 Like
2

I don’t have time right now to do a full review and provide detail comments. The recursive approach works. You may be able to benefit from a little more pattern matching and guard clauses to remove some of the if/else.

Given that your processing complete lists and don’t need early exit, you would be able to accomplish the same thing with Enum.reduce. Also, you may want to look at update_in with multi clause anonymous functions. Something like (random example):

update_in(state, [:key, var], fn 
  nil -> [some_val]  # no entry yet, so return a list
  list -> [some_val | list]   # already has a list for this entry, so prepend to the list
end)

Of course, you can use Map.update too for 1 level of depth. However, if you have nested maps, update_in works great. If you don’t know about multi clause anoyn functions, you should look them up.

1 Like
3 
defmodule Merge do
  def merge(left, right),
    do: Map.merge(to_map(left), to_map(right), &resolve_conflict/3) |> Map.values

  defp to_map(list), do: for item <- list, into: %{}, do: {key(item), item}

  defp key(%{action: action, legal_inventories: legal_inventories}),
    do: {action, Enum.sort(legal_inventories)}

  defp resolve_conflict(_key, %{name: name1} = map1, %{name: name2}),
    do: %{map1 | name: "#{name1} & #{name2}"}
end
3 Likes
4

Wow, I knew my code was overly complex given the relatively simple problem and incredibly great Elixir language… that’s why I posted the request for help.

I never would have dreamed to convert both lists to a map like that (with the key being a tuple representing the composite key) then use Map.merge (with resolve_conflict) to do the dirty work and Map.values to remove the keys and just return the transformed list. I read about Map.merge but felt it wouldn’t work for me because I had lists. It’s very hard to shake the imperative mindset.

Many thanks for such a concise and generic solution, really appreciate it and really hope it helps others in the future!!

2 Likes
5

Yes, map is the way to go. I saw that immediately when I first looked that the problem. Maps have pretty quick access speed. Lists are pretty slow when your accessing anything but the head the list since they are implemented as linked lists.