Remove values from list of nested maps inside a map

script · August 1, 2019, 1:27pm

I have this map that contains a list of maps inside and I want to remove specific values from this.

%{
 "id" => 1   
"rxs" => [
%{
  "antibiotic" => %{       
    "id" => 6042,
    "is_active" => true,
    "name" => "Christy",
    
  },      
  "antibiotic_prescription_changed_to" => %{
    "antibiotic" => %{       
      "id" => 6042,
      "is_active" => true,
      "name" => "Christy",
     
    },       
    "antibiotic_prescription_changed_to" => %{
      "antibiotic" => %{      
        "id" => 6042,
        "is_active" => true,
        "name" => "Christy",
     
      },        
      "antibiotic_prescription_changed_to" => nil,
      "date_end" => nil,
      "date_prescribed" => nil,
      "date_start" => "2019-03-11",
      "dose" => "10mg/mL",
      "encounter" => nil,
      "frequency" => nil,
      "id" => 5751,
      "modified_at" => "2019-08-01T13:04:09",
      "modified_by" => nil,
      "notes" => nil,
      "prescribed_before_admission" => false,
      "prescribing_clinician" => nil,
      "prophylaxis" => false,
      "prophylaxis_indications" => [],
      ...
    },
    "date_end" => nil,
    "date_prescribed" => nil,
    "date_start" => "2019-03-11",
    "dose" => nil,
    "id" => 5752,
    "modified_at" => "2019-08-01T13:04:09",
    "modified_by" => nil,
    "notes" => nil,
    "prescribed_before_admission" => false,
    "prophylaxis" => false,
    "prophylaxis_indications" => []
  },
  "date_end" => nil,
  "date_prescribed" => nil,
  "date_start" => "2019-03-11",
  "dose" => "2 pills a day",
  "encounter" => %{
    "admission_date" => "2017-01-01T00:00:00Z",
    "id" => 3636,
    "is_active" => true,
    "notes" => "A9Q1KGKq",
    "patient_id" => 4862
  },
  "frequency" => nil,
  "id" => 5753,
  "modified_at" => "2019-08-01T13:04:09",
  "modified_by" => nil,
  "notes" => nil,
  "prescribed_before_admission" => false,
  "prescribing_clinician" => nil,
  "prophylaxis" => false,
  "prophylaxis_indications" => [],
  "route" => nil
}
],
"schema_version" => 10

}

I want to remove all the key-value pairs of id and modified_at from antibiotic_prescription_changed_to. It is deeply nested and may contain multiple nested antibiotic_prescription_changed_to.

Some generic approach that will remove these values no matter how many nested maps I get.

I am thinking about some recursion function. which will keep looping in these antibiotic_prescription_changed_to and return the final result once it gets nil.

But the main problem I think is removing all these values and then placed them inside the original map.

Any help will be much appreciated

Thanks

dimitarvp · August 1, 2019, 3:18pm

Let’s backtrack a little: how do you get to this map? Is it sent to you verbatim through another API or do you serialise DB records to maps/JSON objects?

chulkilee · August 1, 2019, 4:53pm

But the main problem I think is removing all these values and then placed them inside the original map.

Think it differently - you just transform the input. Basically, you need to 1) walk the data (recursive call or Enum.map/2) and 2) transform as needed (e.g. Map.put/3) and 3) aggregate to the final result (Enum.into/2).

input = %{
  "id" => 1,
  "rxs" => [
    %{
      "antibiotic" => %{"id" => 6042, "name" => "Christy"},
      "antibiotic_prescription_changed_to" => %{
        "antibiotic" => %{"id" => 6042, "name" => "Christy"},
        "antibiotic_prescription_changed_to" => %{
          "antibiotic" => %{"id" => 6042, "name" => "Christy"},
          "antibiotic_prescription_changed_to" => nil,
          "id" => 5751,
          "modified_at" => "2019-08-01T13:04:09"
        },
        "id" => 5752,
        "modified_at" => "2019-08-01T13:04:09",
        "date_start" => "2019-03-11"
      },
      "encounter" => %{"id" => 3636, "notes" => "A9Q1KGKq"},
      "id" => 5753,
      "modified_at" => "2019-08-01T13:04:09",
      "prescribed_before_admission" => false
    }
  ],
  "schema_version" => 10
}

expected_output = %{
  "id" => 1,
  "rxs" => [
    %{
      "antibiotic" => %{"id" => 6042, "name" => "Christy"},
      "antibiotic_prescription_changed_to" => %{
        "antibiotic" => %{"id" => 6042, "name" => "Christy"},
        "antibiotic_prescription_changed_to" => %{
          "antibiotic" => %{"id" => 6042, "name" => "Christy"},
          "antibiotic_prescription_changed_to" => nil
        },
        "date_start" => "2019-03-11"
      },
      "encounter" => %{"id" => 3636, "notes" => "A9Q1KGKq"},
      "id" => 5753,
      "modified_at" => "2019-08-01T13:04:09",
      "prescribed_before_admission" => false
    }
  ],
  "schema_version" => 10
}

defmodule Transformer do
  def scrub(%{"rxs" => rxs} = data),
    do: Map.put(data, "rxs", Enum.map(rxs, &scrub_antibiotic_prescription_changed_to/1))

  defp scrub_antibiotic_prescription_changed_to(
         %{"antibiotic_prescription_changed_to" => %{} = val} = map
       ) do
    updated_val =
      val
      |> Map.drop(["id", "modified_at"])
      |> scrub_antibiotic_prescription_changed_to()

    Map.put(map, "antibiotic_prescription_changed_to", updated_val)
  end

  defp scrub_antibiotic_prescription_changed_to(%{} = map) do
    map
    |> Enum.map(fn {k, v} -> {k, scrub_antibiotic_prescription_changed_to(v)} end)
    |> Enum.into(%{})
  end

  defp scrub_antibiotic_prescription_changed_to(val), do: val
end

^expected_output = Transformer.scrub(input)

–

Update:

Note that it drops id and modified_at only under antibiotic_prescription_changed_to - so the top level id (which is not under antibiotic_prescription_changed_to is preserved. You can tweak that behavior by changing pattern match (condition) and transformation (func body)

peerreynders · August 1, 2019, 4:53pm

You’ve basically described this:

defmodule Demo do
  def drop_deep(map, from, keys) do
    # drop "keys" from nested "from"s
    from_result =
      case Map.fetch(map, from) do
        {:ok, value} when is_map(value) ->
          {:ok, drop_deep(value, from, keys)}

        _ ->
          nil
      end

    # merge updated "from" value
    map_result =
      case from_result do
        {:ok, new_value} ->
          Map.put(map, from, new_value)

        _ ->
          map
      end

    # drop keys on this level
    Map.drop(map_result, keys)
  end

  def update_rxs(rxs) do
    Enum.map(
      rxs,
      &drop_deep(
        &1,
        "antibiotic_prescription_changed_to",
        ["id", "modified_at"]
      )
    )
  end

  def run(data),
    do: Map.update(data, "rxs", [], &update_rxs/1)
end

data = %{
  "id" => 1,
  "rxs" => [
    %{
      "antibiotic" => %{
        "id" => 6042,
        "is_active" => true,
        "name" => "Christy"
      },
      "antibiotic_prescription_changed_to" => %{
        "antibiotic" => %{
          "id" => 6042,
          "is_active" => true,
          "name" => "Christy"
        },
        "antibiotic_prescription_changed_to" => %{
          "antibiotic" => %{
            "id" => 6042,
            "is_active" => true,
            "name" => "Christy"
          },
          "antibiotic_prescription_changed_to" => nil,
          "date_end" => nil,
          "date_prescribed" => nil,
          "date_start" => "2019-03-11",
          "dose" => "10mg/mL",
          "encounter" => nil,
          "frequency" => nil,
          "id" => 5751,
          "modified_at" => "2019-08-01T13:04:09",
          "modified_by" => nil,
          "notes" => nil,
          "prescribed_before_admission" => false,
          "prescribing_clinician" => nil,
          "prophylaxis" => false,
          "prophylaxis_indications" => []
        },
        "date_end" => nil,
        "date_prescribed" => nil,
        "date_start" => "2019-03-11",
        "dose" => nil,
        "id" => 5752,
        "modified_at" => "2019-08-01T13:04:09",
        "modified_by" => nil,
        "notes" => nil,
        "prescribed_before_admission" => false,
        "prophylaxis" => false,
        "prophylaxis_indications" => []
      },
      "date_end" => nil,
      "date_prescribed" => nil,
      "date_start" => "2019-03-11",
      "dose" => "2 pills a day",
      "encounter" => %{
        "admission_date" => "2017-01-01T00:00:00Z",
        "id" => 3636,
        "is_active" => true,
        "notes" => "A9Q1KGKq",
        "patient_id" => 4862
      },
      "frequency" => nil,
      "id" => 5753,
      "modified_at" => "2019-08-01T13:04:09",
      "modified_by" => nil,
      "notes" => nil,
      "prescribed_before_admission" => false,
      "prescribing_clinician" => nil,
      "prophylaxis" => false,
      "prophylaxis_indications" => [],
      "route" => nil
    }
  ],
  "schema_version" => 10
}

result = Demo.run(data)
IO.inspect(result)

$ elixir demo.exs
%{
  "id" => 1,
  "rxs" => [
    %{
      "antibiotic" => %{"id" => 6042, "is_active" => true, "name" => "Christy"},
      "antibiotic_prescription_changed_to" => %{
        "antibiotic" => %{
          "id" => 6042,
          "is_active" => true,
          "name" => "Christy"
        },
        "antibiotic_prescription_changed_to" => %{
          "antibiotic" => %{
            "id" => 6042,
            "is_active" => true,
            "name" => "Christy"
          },
          "antibiotic_prescription_changed_to" => nil,
          "date_end" => nil,
          "date_prescribed" => nil,
          "date_start" => "2019-03-11",
          "dose" => "10mg/mL",
          "encounter" => nil,
          "frequency" => nil,
          "modified_by" => nil,
          "notes" => nil,
          "prescribed_before_admission" => false,
          "prescribing_clinician" => nil,
          "prophylaxis" => false,
          "prophylaxis_indications" => []
        },
        "date_end" => nil,
        "date_prescribed" => nil,
        "date_start" => "2019-03-11",
        "dose" => nil,
        "modified_by" => nil,
        "notes" => nil,
        "prescribed_before_admission" => false,
        "prophylaxis" => false,
        "prophylaxis_indications" => []
      },
      "date_end" => nil,
      "date_prescribed" => nil,
      "date_start" => "2019-03-11",
      "dose" => "2 pills a day",
      "encounter" => %{
        "admission_date" => "2017-01-01T00:00:00Z",
        "id" => 3636,
        "is_active" => true,
        "notes" => "A9Q1KGKq",
        "patient_id" => 4862
      },
      "frequency" => nil,
      "modified_by" => nil,
      "notes" => nil,
      "prescribed_before_admission" => false,
      "prescribing_clinician" => nil,
      "prophylaxis" => false,
      "prophylaxis_indications" => [],
      "route" => nil
    }
  ],
  "schema_version" => 10
}
$

Eiji · August 1, 2019, 6:08pm

@script This is good question. Depends on use case you may remove it at query language (like GraphQL or SQL) level. In such case you do not need to write any Elixir code.

@chulkilee Not bad, but after few simple changes your code could be easier to maintain. Similar goes to @peerreynders, but his code is incredibly long.

Here is my code:

defmodule Example do
  # Nothing is better than easily configurable code :-)
  @key_to_find "antibiotic_prescription_changed_to"
  @keys_to_drop ["id", "modified_at"]

  # We are starting from "rxs" field
  def sample(%{"rxs" => rxs} = data), do: %{data | "rxs" => do_sample(rxs)}

  # First of all we need to make sure that we supports lists
  # You can use &Stream.map/2 as well in case of bigger lists
  # If order does not matter you can also use Flow instead of Stream
  defp do_sample(data) when is_list(data), do: Enum.map(data, &do_sample/1)

  # We are only looking for maps with specified key
  # Some values may be nil, so we need to add extra is_map/1 check
  defp do_sample(%{@key_to_find => value} = data) when is_map(value) do
    # Note "|> do_sample()" at end of line is not needed
    # if you would uncomment rest of code
    updated_value = value |> Map.drop(@keys_to_drop) |> do_sample()
    # Please make sure to remove _ (underscore) character
    # in case you would like to uncomment other code
    _updated_map = %{data | @key_to_find => updated_value}
    # Nested call for other keys if needed
    # :maps.map(&other_map_keys/2, updated_map)
  end

  # Extra clause for maps without @key_to_find with map data
  # This is useful if other nested maps could contain also @key_to_find
  # defp do_sample(data) when is_map(data),
  #   do: :maps.map(&other_map_keys/2, data)

  # Simply do not change data which is not list or map
  defp do_sample(data), do: data

  # Helper function used in &:maps.map/3
  # This function only changes value and key is not touched
  # We are ignoring key as it's not needed in to change value
  # defp other_map_keys(_key, value), do: do_sample(value)
end

data = %{
  "id" => 1,
  "rxs" => [
    %{
      "antibiotic" => %{
        "id" => 6042,
        "is_active" => true,
        "name" => "Christy"
      },
      "antibiotic_prescription_changed_to" => %{
        "antibiotic" => %{
          "id" => 6042,
          "is_active" => true,
          "name" => "Christy"
        },
        "antibiotic_prescription_changed_to" => %{
          "antibiotic" => %{
            "id" => 6042,
            "is_active" => true,
            "name" => "Christy"
          },
          "antibiotic_prescription_changed_to" => nil,
          "date_end" => nil,
          "date_prescribed" => nil,
          "date_start" => "2019-03-11",
          "dose" => "10mg/mL",
          "encounter" => nil,
          "frequency" => nil,
          "id" => 5751,
          "modified_at" => "2019-08-01T13:04:09",
          "modified_by" => nil,
          "notes" => nil,
          "prescribed_before_admission" => false,
          "prescribing_clinician" => nil,
          "prophylaxis" => false,
          "prophylaxis_indications" => []
        },
        "date_end" => nil,
        "date_prescribed" => nil,
        "date_start" => "2019-03-11",
        "dose" => nil,
        "id" => 5752,
        "modified_at" => "2019-08-01T13:04:09",
        "modified_by" => nil,
        "notes" => nil,
        "prescribed_before_admission" => false,
        "prophylaxis" => false,
        "prophylaxis_indications" => []
      },
      "date_end" => nil,
      "date_prescribed" => nil,
      "date_start" => "2019-03-11",
      "dose" => "2 pills a day",
      "encounter" => %{
        "admission_date" => "2017-01-01T00:00:00Z",
        "id" => 3636,
        "is_active" => true,
        "notes" => "A9Q1KGKq",
        "patient_id" => 4862
      },
      "frequency" => nil,
      "id" => 5753,
      "modified_at" => "2019-08-01T13:04:09",
      "modified_by" => nil,
      "notes" => nil,
      "prescribed_before_admission" => false,
      "prescribing_clinician" => nil,
      "prophylaxis" => false,
      "prophylaxis_indications" => [],
      "route" => nil
    }
  ],
  "schema_version" => 10
}

Example.sample(data)

Note: Of course I could use {} = value instead of when is_map(value), but nested pattern matching does not looks really readable especially for newbies.

As you can see my code is configurable and using attributes we can reduce code a lot. My function clauses fits in just single line. My version is also easiest to change in case input data would be modified in future which happens often especially dev side (for example database migrations).

chulkilee · August 1, 2019, 6:44pm

Great! It always takes multiple iterations to get “good” code - so thanks for taking that part

Some notes

Avoid do_... naming - and use meaningful names
Instead of one function for two pattern match, we may use separate functions; see below:

# two things in one func
defp scrub(%{@key_to_find => value} = data) when is_map(value) do
  updated_value = value |> Map.drop(@keys_to_drop) |> scrub()
  %{data | @key_to_find => updated_value}
end
# other scrub

# separate funcs

defp scrub(%{@key_to_find => value} = data), do: %{data | @key_to_find => drop_keys(value)}
# other scrub

defp drop_keys_and_scrub(data) when is_map, do: value |> Map.drop(@keys_to_drop) |> scrub()
defp drop_keys_and_scrub(data), do: scrub(data)

On extensibility… I prefer not to create generic utility module unless I found it’s repeated at least 3 places. I found it’s usually better to code more verbose and less generic until all I know what actually I need. For example, the author may need to apply more sophisticated scrubbing based on the context - which defeats the “general-purpose” solution

Also. if it is super performance-critical part and handles a large data - NIF would be a better option - like Using Rust to Scale Elixir for 11 Million Concurrent Users

Eiji · August 1, 2019, 11:53pm

That’s interesting. I’m not exactly sure based on which code I started to use such naming. I was a bit curious and want to check most important git repository for Elixir world … Yes, I mean elixir-lang/elixir:

$ git clone https://github.com/elixir-lang/elixir.git
$ cd elixir
$ grep -r "defp do_" | wc -l
303

Of course I’m also using meaningful naming. Here I have used Example and sample naming as they are just easy to remember, so it’s my default in case author would not provide an example code with his own naming.

Yes, but it depends on use case. Also personally I do not like a recursion called in other function. When code grows it may become hard to understand - again especially for newbies. For sure I believe that after “quick remind” both me and you could get back to this code and still understand it, but good code in my opinion should not require a “quick remind”. I’m always thinking if I would be able to read my code after few years without any bigger pause. In bigger business logic such recursion could be much harder to find.

When reading this TDD goes to my mind first. Maybe it’s about my freelancer experience as I mostly worked for start-ups in which things changes rapidly. Also maybe I used wrong word. By generic I mean that my code is easy to change and it does not require much work on it (see commented code example). I mean that at same time allow to work in different schema scenario and also do not conflict with current one i.e. A and B - not A over B or B over A.

In 3rd line you assume that rxs would always store list. Of course it would probably want be changed, but literally everything could happen. When I’m working with data structures I’m always thinking how input could be modified and how much it would affect my code. In dev environment when client really often changes mind you have 2 ways:

Create generic ecto schema files
I always require full vision of project. Not only how it looks like, but also how it’s planned to change in future. Therefore I’m often working longer on schema files making it bigger and less changed in future (except changesets).
Deal with lots of migrations
In such case you can’t assume if has_many will always be same. It could be changed to belongs_to or be removed. More typical scenario is that except recursive structure other relations could somehow depend on that schema. Then again take a look at my comments. Almost all you need to do is to just uncomment few lines.

I did not know if this input would not change in future. That’s why first of all I have created full code and later commented some parts.

That’s true, but looking on both my and your code example I do not see a problem as Map.drop/2 is used in one place only and there we can call some extra Map transformations. I also though about it.

chulkilee · August 2, 2019, 12:10am

script · August 2, 2019, 2:22pm

Thanks for your solution. This is what I need.

script · August 2, 2019, 2:23pm

Thank you.

script · August 2, 2019, 2:24pm

Thank you for your answer.