rmoretto
Hello!
So I currently have the following problem: I want to take a map and “sanitize” all the integers fields. Basically verifying if that field value is really an int and replace the parsed value in the map, furthermore, if the verified value is not an valid int it should save nil to the field, for example:
Input Map
%{
int_a: "1",
int_b: "not_a_int!"
%}
Output Map:
%{
int_a: 1,
int_b: nil
%}
On the system that I am currently working I know that all the integers fields in the map will start with the int_ prefix (important to say that I don’t have full access to the struct of the map, so I can’t write a function that uses a specific field) so I wrote the following code:
def parse_map_int_fields(map) do
ints_map =
map
# Filter all the Integers field
|> Enum.filter(fn {k, _v} -> Atom.to_string(k) |> String.starts_with?("int_") end)
# Remove all fields that are nil or ints
|> Enum.filter(fn {_k, v} -> !is_integer(v) and !is_nil(v) end)
# Parse the values to int, returning nil if invalid
|> Enum.map(fn {key, val} ->
case Integer.parse(val) do
:error ->
{key, nil}
{parsed, _} ->
{key, parsed}
end
end)
|> Map.new()
# Merge the original map with the "sanitized" map
Map.merge(map, ints_map)
end
Another point is that this code will be ran in a lot of maps (5 to 10 millions maps) with on average 15 fields per map, so I have two questions:
- Most important, can this code be simplified? From my perspective it look a little bit convoluted, I would appreciate any tips!
- How one would optimize this function? I have wrote a simple benchmark using Benchee that gave me the following results:
Operating System: Linux
CPU Information: AMD Ryzen 7 2700X Eight-Core Processor
Number of Available Cores: 16
Available memory: 15.63 GB
Elixir 1.11.2
Erlang 23.2.3
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 7 s
Benchmarking parse_map_int_fields...
Name ips average deviation median 99th %
parse_map_int_fields 857.59 K 1.17 μs ±2149.16% 1.01 μs 2.12 μs
Code used for the benchmark:
defmodule Sanitizer do
def parse_map_int_fields(map) do
ints_map =
map
|> Enum.filter(fn {k, _v} -> Atom.to_string(k) |> String.starts_with?("int_") end)
|> Enum.filter(fn {_k, v} -> !is_integer(v) and !is_nil(v) end)
|> Enum.map(fn {key, val} ->
case Integer.parse(val) do
:error ->
{key, nil}
{parsed, _} ->
{key, parsed}
end
end)
|> Map.new()
Map.merge(map, ints_map)
end
end
test_map = %{
int_a: "1",
int_b: "not_a_int!",
str_a: "This is a string field",
str_b: "Another string field",
fl_a: 0.0
}
Benchee.run(%{
"parse_map_int_fields" => fn -> Sanitizer.parse_map_int_fields(test_map) end
})
Which isn’t all that bad for my use case, but in the interest of learning I would like to know if something could be done different.
Thanks to all! ![]()
Trending in Questions
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming










First Post!- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
kokolegorille
You could simplify in one pass…
In fact, You could probably do all in one pass. And You should not filter, if later You will merge the filtered {k, v}
Maybe something like this, one iteration only…
You can even replace with a private function this part.
You should not hesitate to split your pipeline into small, composable functions.
Most Liked
gregvaughn
I haven’t benchmarked, but you could do something with a for comprehension
rmoretto
Thanks people for all the responses!
Indeed a lot of way to approach the problem, I will probably use the @amnu3387 response just because is a little bit faster, but as Eric said, is just a matter of taste, and all other options would be completely fine.
Thanks again for the help!
The benchmark results were:
And the code used for the benchmark:
eksperimental
you can optimize it even more:
def maybe_reshape(k, v) when is_atom(k) and is_binary(v) doLast Post!
dimitarvp
Naive solution on a first try:
Try in
iex:Haven’t measured but
forcomprehensions with filters and thereduceoption have always been fairly fast in my experience.Maybe if you make a public GitHub repo then we can make a benchmarking arena.