Shaping flat maps into nested maps with changing data inputs

Questions / Help
, , ,
1

I am working to shape request parameters that are flat into nested maps to work with changesets in Ecto. The intention is to only update the rows based on the keys passed on the parameters so when some keys are absent it should not result in a nil

%{
  "username" => params["username"],
"public_profile" => %{"birthday" => (if params["birthday"], do: params["birthday"]|> DateTime.from_unix!() |> DateTime.to_naive()),
}

So currently my stop gap has been to introduce a helper fn that purges all nil values at each hierarchy layer of the map.

defp purge_nil(map), do: map |> Enum.reject(fn {_, v} -> is_nil(v) end) |> Map.new()

That I employ like so:

%{
      "username" => params["username"],
      "public_profile" => %{"birthday" => (if params["birthday"], do: params["birthday"]|> DateTime.from_unix!() |> DateTime.to_naive()),
                           "occupation" => params["occupation"]} |> purge_nil(),
       |> purge_nil()

Which definitely is unwieldy, how might I approach this data transformation from a place of greater elegance

2

Assuming that you need to parse form data into some of the fields of a User Ecto model:

defmodule MyApp.User do
  use Ecto.Schema

  @required_fields [:username] # and others

  schema "users" do
    field(:username, :string)
    field(:birthday, :datetime)
    # ... other fields here
  end

  def from_params(%{"username" => username, "public_profile" => %{"birthday" => birthday}}) do
    %{username: username, birthday: birthday}
  end

  def from_params(%{"username" => username) do
    %{username: username}
  end

  def changeset(struct \\ %__MODULE__{}, data) when is_map(data) do
    struct
    |> cast(params, @required_fields)
  end
end

Then you can do e.g. in your controller:

  user = params |> User.from_params() |> User.changeset()

And take it from there.

You don’t need to introduce nil values and then remove them. You can just make separate variants of how to construct a user depending on the shape of the input data.

3

Here you go:

defmodule Example do
  # list of keys we want to add as is to the root map without parsing
  @flat_keys ~w[occupation username]

  # a simple reduce params call over empty map as accumulator
  def sample(params) when is_map(params), do: Enum.reduce(params, %{}, &sample/2)

  # when some key does not exists it does then it simply does not apply in this example
  # however if you expect a nil value to be passed in params you can use this simple patter matching
  defp sample({_key, nil}, acc), do: acc

  # simple check if key exists within a flat_keys list
  defp sample({key, value}, acc) when key in @flat_keys, do: Map.put(acc, key, value)

  # in any other case
  defp sample({key, value}, acc) do
    # fetch a custom path and parsed value
    {path, parsed_value} = sample(key, value)
    # map path list so in case nested structure was not yet created
    key_path = Enum.map(path, &Access.key(&1, %{}))
    # put_in would automatically add an empty map as value and parsed value as a leaf
    put_in(acc, key_path, parsed_value)
  end

  # below is a separate place for your custom parsing logic
  defp sample("birthday", birthday) when is_integer(birthday) do
    naive = birthday |> DateTime.from_unix!() |> DateTime.to_naive()
    # look that path can also have one key, so you can parse a root key's value as well
    {~w[public_profile birthday], naive}
  end
end

iex> Example.sample(%{"birthday" => 0, "occupation" => "NEET", "username" => "John Doe"})
%{
  "occupation" => "NEET",
  "public_profile" => %{"birthday" => ~N[1970-01-01 00:00:00]},
  "username" => "John Doe"
}

iex> Example.sample(%{"username" => "John Doe"})
%{"username" => "John Doe"}

You can even easily change String to Atom keys simply calling String.to_atom/1 on a key variable used in Map.put/3 call and change path from list of String into a list of Atom using ~w[key1 key2 key3]a notation instead.

Helpful resources:

  1. Access.key/2
  2. Enum.map/2
  3. Enum.reduce/3
  4. Kernel.is_integer/1
  5. Kernel.is_map/1
  6. Kernel.sigil_w/2
  7. Map.put/3
4

Hmm this approach is alright for flattening a data structure but also encounters limitations as the number and combination of fields possible increases especially when you are allowing the input requests to change only rows/keys they want to change/replace either in the nested associations or embeds

5

This is a better approach, I believe this key/value tuple construct is quite powerful in expressing the nested structure

{~w[public_profile birthday], naive}

And extending this through

when key in @public_profile_keys

Allows us to leverage guards and pattern matching to great effect.

Would creating such a module for each request within the API be considered bloated or is there a simple interface through which can access an abstraction of such a parser

6

Nothing stops you from writing one generic module and each module for every of your APIs.

Also you can use metaprogramming like:

defmodule Example do
  use MyLib, flat_keys: @flat_keys ~w[occupation username]

  def parse("birthday", birthday) when is_integer(birthday) do
    naive = birthday |> DateTime.from_unix!() |> DateTime.to_naive()
    {~w[public_profile birthday], naive}
  end
end

Another way is to use an existing library like cozy_params:

In above topic I have suggested an interesting ideas for source and sources options. Those are already approved and once implemented it would suit your case perfectly.

See also related GitHub issue:

1 Like
7

Metaprogramming is a neat suggestion will definitely employ that, the issue the user was facing has a deep similarity with my problem space. Thanks for suggesting this great extension to the library will keep my eye out for it