TransformMap - Deeply Nested Maps into flat maps or 2 dimensional array. Export any map to CSV, XLSX or JSON

pedromvieira

Transform Elixir Deeply Nested Maps into flat maps or 2 dimensional array. Export any map to CSV, XLSX or JSON.

This will help in converting any deep nested map into a flap map or array to export it. Tested with huge maps and several thousands lines.

Please give your feedback.

https://hex.pm/packages/transform_map

https://github.com/pedromvieira/transform_map

Usage:

  iex> TransformMap.multiple_shrink(map, "|", true, true)

  iex> TransformMap.multiple_expand(shrink_map, "|", true)

  iex> TransformMap.multiple_to_array(map, "|", true, true)

  iex> TransformMap.multiple_keys(shrink_map, true)

  iex> TransformMap.multiple_to_array(map, "|", true, true) |> TransformMap.Export.to_csv("test.csv", true)

  iex> TransformMap.multiple_to_array(map, "|", true, true) |> TransformMap.Export.to_xlsx("test.xlsx", true)

  iex> map |> TransformMap.Export.to_json("test.json", true)

  iex> "temp/test.csv" |> TransformMap.Export.to_gzip()

2 comments

#library #maps #export #transform_map

3 1729 2

2018-04-27 17:20:40 UTC

Showing Posts 1 to 2

Show Best Posts
Show All (oldest first)
Show All (newest first)

OvermindDL1

Can you give some examples of, say, at least 4-deep nested maps being converted to a csv as well as some other examples? Also it does proper csv escaping and xlsx and so forth? Is there a way to decorate the xlsx nodes with formatting data for better displaying to the users?

Post #1

pedromvieira

It uses some libraries to do properly escape and coding. By default it’s all UTF-8 with basic escaping, but you can use your own functions.
Give it a try!

External Libraries:

JSON - exjsx | Hex
CSV - csvlixir | Hex
XLSX - elixlsx | Hex

And for performance:

PARALLEL STREAM - parallel_stream | Hex

Example:

map = [
  %{
    "action" => "sent",
    "data" => %{
      "channel" => "email",
      "message" => %{
        "campaign" => 1,
        "editor_id" => 1,
        "group" => 2,
        "list_type" => "regular",
        "schedule" => 4,
        "shortener" => "default",
        "target_data" => %{
          "birthday" => "1982-06-09T15:00:00",
          "businessunit" => "Board",
          "locale" => "eng",
          "location_data" => %{
            "city" => "São Paulo",
            "city_data" => %{"mayor" => "John Snow", "population" => 30000000}
          },
          "nome" => "Pedro",
          "role" => "CTO"
        }
      },
      "provider" => "sendgrid"
    },
    "id" => 8,
    "inserted_at" => "2018-04-02 18:07:52.260159",
    "key" => "cGhpc2h4fDF8NHwyfHJlZ3"
  }
]

shrink_map = TransformMap.multiple_shrink(map, "|", true, true)
[
  %{
    "action" => "sent",
    "data|channel" => "email",
    "data|message|campaign" => 1,
    "data|message|editor_id" => 1,
    "data|message|group" => 2,
    "data|message|list_type" => "regular",
    "data|message|schedule" => 4,
    "data|message|shortener" => "default",
    "data|message|target_data|birthday" => "1982-06-09T15:00:00",
    "data|message|target_data|businessunit" => "Board",
    "data|message|target_data|locale" => "eng",
    "data|message|target_data|location_data|city" => "São Paulo",
    "data|message|target_data|location_data|city_data|mayor" => "John Snow",
    "data|message|target_data|location_data|city_data|population" => 30000000,
    "data|message|target_data|nome" => "Pedro",
    "data|message|target_data|role" => "CTO",
    "data|provider" => "sendgrid",
    "id" => 8,
    "inserted_at" => "2018-04-02 18:07:52.260159",
    "key" => "cGhpc2h4fDF8NHwyfHJlZ3"
  }
]

array = TransformMap.multiple_to_array(map, "|", true, true)
[
  ["action", "data|channel", "data|message|campaign", "data|message|editor_id",
   "data|message|group", "data|message|list_type", "data|message|schedule",
   "data|message|shortener", "data|message|target_data|birthday",
   "data|message|target_data|businessunit", "data|message|target_data|locale",
   "data|message|target_data|location_data|city",
   "data|message|target_data|location_data|city_data|mayor",
   "data|message|target_data|location_data|city_data|population",
   "data|message|target_data|nome", "data|message|target_data|role",
   "data|provider", "id", "inserted_at", "key"],
  ["sent", "email", 1, 1, 2, "regular", 4, "default", "1982-06-09T15:00:00",
   "Board", "eng", "São Paulo", "John Snow", 30000000, "Pedro", "CTO",
   "sendgrid", 8, "2018-04-02 18:07:52.260159", "cGhpc2h4fDF8NHwyfHJlZ3"]
]

array |> TransformMap.Export.to_csv("test.csv", true)
{:ok, ".../temp/test.csv.gz"}

CSV
action,data|channel,data|message|campaign,data|message|editor_id,data|message|group,data|message|list_type,data|message|schedule,data|message|shortener,data|message|target_data|birthday,data|message|target_data|businessunit,data|message|target_data|locale,data|message|target_data|location_data|city,data|message|target_data|location_data|city_data|mayor,data|message|target_data|location_data|city_data|population,data|message|target_data|nome,data|message|target_data|role,data|provider,id,inserted_at,key
sent,email,1,1,2,regular,4,default,1982-06-09T15:00:00,Board,eng,São Paulo,John Snow,30000000,Pedro,CTO,sendgrid,8,2018-04-02 18:07:52.260159,cGhpc2h4fDF8NHwyfHJlZ3

Post #2

— All posts loaded —