Babel - Data transformations made easy

wolf4earth

Babel was born out of a desire to simplify non-trivial data transformation pipelines. To focus on the “happy path” instead of having to write a bunch of boilerplate error handling code.

But don’t listen to me, take a look for yourself:

pipeline =
  Babel.begin()
  |> Babel.fetch(["some", "nested", "path"])
  |> Babel.map(Babel.into(%{atom_key: Babel.fetch("string-key")}))

data = %{
  "some" => %{
    "nested" => %{
      "path" => [
        %{"string-key" => :value2},
        %{"string-key" => :value2},
        %{"string-key" => :value2}
      ]
    }
  }
}

Babel.apply(pipeline, data)
=> {:ok, [
   %{atom_key: :value1},
   %{atom_key: :value2},
   %{atom_key: :value3}
]}

Since you’ll most likely build non-trivial transformation pipelines with Babel - which can fail at any given step - Babel ships with elaborate error reporting:

Error Reporting

pipeline =
  Babel.begin()
  |> Babel.fetch(["some", "nested", "path"])
  |> Babel.map(Babel.into(%{atom_key: Babel.fetch("string-key")}))

data = %{
  "some" => %{
    "nested" => %{
      "path" => [
        %{"unexpected-key" => :value1},
        %{"unexpected-key" => :value2},
        %{"unexpected-key" => :value3}
      ]
    }
  }
}

Babel.apply!(pipeline, data)

Which will produce the following error

** (Babel.Error) Failed to transform data: [not_found: "string-key", not_found: "string-key", not_found: "string-key"]

Root Cause(s):
1. Babel.Trace<ERROR>{
  data = %{"unexpected-key" => :value1}

  Babel.fetch("string-key")
  |=> {:error, {:not_found, "string-key"}}
}
2. Babel.Trace<ERROR>{
  data = %{"unexpected-key" => :value2}

  Babel.fetch("string-key")
  |=> {:error, {:not_found, "string-key"}}
}
3. Babel.Trace<ERROR>{
  data = %{"unexpected-key" => :value3}

  Babel.fetch("string-key")
  |=> {:error, {:not_found, "string-key"}}
}

Full Trace:
Babel.Trace<ERROR>{
  data = %{"some" => %{"nested" => %{"path" => [%{"unexpected-key" => :value1}, %{"unexpected-key" => :value2}, %{"unexpected-key" => :value3}]}}}

  Babel.Pipeline<>
  |
  | Babel.fetch(["some", "nested", "path"])
  | |=< %{"some" => %{"nested" => %{"path" => [%{"unexpected-key" => :value1}, %{...}, ...]}}}
  | |=> [%{"unexpected-key" => :value1}, %{"unexpected-key" => :value2}, %{"unexpected-key" => :value3}]
  |
  | Babel.map(Babel.into(%{atom_key: Babel.fetch("string-key")}))
  | |=< [%{"unexpected-key" => :value1}, %{"unexpected-key" => :value2}, %{"unexpected-key" => :value3}]
  | |
  | | Babel.into(%{atom_key: Babel.fetch("string-key")})
  | | |=< %{"unexpected-key" => :value1}
  | | |
  | | | Babel.fetch("string-key")
  | | | |=< %{"unexpected-key" => :value1}
  | | | |=> {:error, {:not_found, "string-key"}}
  | | |
  | | |=> {:error, [not_found: "string-key"]}
  | |
  | | Babel.into(%{atom_key: Babel.fetch("string-key")})
  | | |=< %{"unexpected-key" => :value2}
  | | |
  | | | Babel.fetch("string-key")
  | | | |=< %{"unexpected-key" => :value2}
  | | | |=> {:error, {:not_found, "string-key"}}
  | | |
  | | |=> {:error, [not_found: "string-key"]}
  | |
  | | Babel.into(%{atom_key: Babel.fetch("string-key")})
  | | |=< %{"unexpected-key" => :value3}
  | | |
  | | | Babel.fetch("string-key")
  | | | |=< %{"unexpected-key" => :value3}
  | | | |=> {:error, {:not_found, "string-key"}}
  | | |
  | | |=> {:error, [not_found: "string-key"]}
  | |
  | |=> {:error, [not_found: "string-key", not_found: "string-key", not_found: "string-key"]}
  |
  |=> {:error, [not_found: "string-key", not_found: "string-key", not_found: "string-key"]}
}

10 comments

#library #pipeline

8 1041 10

2024-05-17 22:09:28 UTC

Most Liked

Show Best Posts
Show All (oldest first)
Show All (newest first)

wolf4earth

If you’re wondering whether or not Babel is production-ready: we’ve been using a pre-release version for nearly a year at this point at work in production, and it made external API integrations a lot easier and smoother.

Post #1

wolf4earth

What I’m taking away from this, is that more examples would be helpful and a comparison against libraries like Ecto.

Post #9

kasvith

babel can be misinterpreted as the popular js transpiler

Post #2

Last Post!

Eiji

This one was only for the example you shared as you said it’s more complex one it’s used in production, so I have tried to re-produce it.

I didn’t have a problem with that at all, but who am I to speak about it when Gentoo was my first Linux distribution.

Perhaps you mean cases like Map.fetch/2 returning :error atom? It could be solved with a simple wrapper function.

def map_fetch(map, key) do
  case Map.fetch(map, key) do
    {:ok, result} -> {:ok, result}
    :error -> {:error, %Wrapper.MapKeyError{key: key, map: map}}
  end
end

All Elixir core features including debugging is more than enough for me and I rarely need to write any wrapper. As said if Elixir core is not enough at most I have used ecto.

Ignore that, I got something wrong when quickly browsing documentation.

That’s a good point … for a Rust developers.

I’m writing pipelines in pure Elixir supporting not single, but all “happy paths” and either “let it default” to nil in case some part of data is expected to be nullable or let it fail when it’s not supported. Especially the “let it fail” is Erlang and Elixir concept, so speaking generally about “happy paths” have not sense in general context in such languages. Therefore at the very end the code is very specific to your needs. I believe that your package may therefore be or not be used even in very similar cases. Just speaking it’s a good for “Rust” was 50-50 for fun.

Sounds like latest Elixir improvements like dbg macro, improved error messages and using part of code in error output. I guess that I personally would be more interested in such package before mentioned improvements, but at the end I would stop use said package after that improvements would be added.

Again who am I to speak about that when in many answers I gave few possible solutions where the biggest one is based only on the pattern-matching. I have no idea, but for some reason I like how things works now.

Oh, at first I did not expanded it, so I did not saw it. Hmm …

Too long! Shorten it into 5 words!

More seriously … it’s really long and maybe that’s because it somehow reminds me a terrible long stacktrace in JVM-based languages. I’m exhausted by just seeing and XX more at the bottom of already too long stacktrace. Again I like how things works now, so I’m not sure if I can see a real-world usage in my personal case.

Why you mention control? You have only control if your app is generating the data, right? As said for all those years I have used pure Elixir and at most ecto to cover almost all if not all cases of casting a data from for example JSON API response.

What? You are developing the “shape” as same as you are creating structs. Regardless if data have proper format or not you simply pass "audioCollection" value to changeset function and in worst case you use some mapper like the default one supported by ecto i.e. @field_source_mapper mentioned previously.

Yeah, I guess that I did not get non-ecto use cases.

I can see that posting:

Data transformations made easy.

without explaining what exactly use-cases you mean is really confusing. That makes me only more interested to hear “your story”.

Maybe answering such questions could help you describe your point of view more easily:

Could you please describe uncommon data sources from your use cases?
Could you please describe by example where casting data by ecto starts causing trouble?
Why things like @field_source_mapper are not enough for a data with different structure?

Edit: When writing all of this I got some idea. If it’s not “something like” JSON API that have a well known structure then the source could have a dynamic structure. Ok, but how to parse a dynamic structure? How about creating a complex html form for dynamically querying data of unknown for the app structure (but known by user)? That’s interesting, but how such data would be used? How about using dynamic rules to fetch dynamic data in order to present it in a predictable format like a graph? Am I going into the right direction or by any change I have messed it up completely? Maybe I think about it too much and there is much simpler use case …