Map, filter and reduce

Hi there,

I want to get the average height of a given list of people. Here is the associated code :

    people = [%{name: 'Mary', height: 160},
                %{name: 'Paul', height: 180},
                %{name: 'Hugo'}]

    heights = Enum.map(people, fn person -> Map.get(person, :height) end)
    heights = Enum.filter(heights, fn h -> h != nil end)

    if length(heights) > 0, do: (
        Enum.reduce(heights, 0, fn (h, total) -> total + h end) / length(heights)
    )

Is there a better way to do this? I would like to know if there is a way to filter a list of maps by a given key (here, the height property).

Thanks,
Didier

2 Likes

I’d go for roughly the following as an avg function:

def avg(list)
  list
  |> Enum.reduce({0, 0}, &avg/2)
  |> avg_finalize
end

defp avg(x, {sum, count}), do: {sum + x, count + 1}
defp avg_finalize({sum, count}), do: sum / count

The cool thing about this is, that you only need to iterate a single time over the input list.

Currently it would fail on non-numeric input, but you can easily define appropriate guards. Adding one which does simply ignore nils should be easy.

And as a small rule of thumb, when you start chaining from one Enum-function into another, you might think about using Stream instead to reduce the number of iterations over the complete input.

But this really depends on the size of your input, if your input is short enough and the Enum-chains also, then Stream might take much more time.

5 Likes

Applying @NobbZ’s approach to your specific situation, I’d be looking at something like:

people = [%{name: 'Mary', height: 160},
                %{name: 'Paul', height: 180},
                %{name: 'Hugo'}]

{total_height, count_people_with_height} =  
  Enum.reduce(people, {0,0}, fn %{height: height}, {sum, count} -> {sum + height, count + 1}
                                                _, {sum, count} -> {sum, count}
end)

IO.puts total_height / count_people_with_height
5 Likes

You are almost at the stage where it is easier to write the loop directly and not use Enum. :wink:

11 Likes

You say “loop” and that makes me wonder. I don’t see a good way to do it in one pass with a list comprehension. I can do it with recursion, but I prefer Enum.reduce

defmodule Test do
  def avg(maps), do: avg(maps, {0,0})
  def avg([], {total, count}) when count > 0, do: total / count
  def avg([%{height: height} | tail], {total, count}), do: avg(tail, {total + height, count + 1})
  def avg([_ | tail], acc), do: avg(tail, acc)
end

people = [%{name: 'Mary', height: 160},
          %{name: 'Paul', height: 180},
          %{name: 'Hugo'}]
IO.puts Test.avg(people)
2 Likes

My professor for functional programming used to say, that “loop” is just another word for recursion :wink:

6 Likes

Thanks for your replies! I really appreciate ! I’m a totally noob with functional programming and elixir. This is far different from traditional OOP…

2 Likes

Hi,
I am at very nascent stage in elixir , was wondering how can we achieve below case scenario in elixir

var doctors=[
    { doctorNumber: "#9",  playedBy: "Christopher Eccleston", yearsPlayed: 1 },
    { doctorNumber: "#10", playedBy: "David Tennant",         yearsPlayed: 6 },
    { doctorNumber: "#11", playedBy: "Matt Smith",            yearsPlayed: 4 },
    { doctorNumber: "#12", playedBy: "Peter Capaldi",         yearsPlayed: 1 }
] 

I only require to get only two columns out of three with new column names !

doctors = doctors.map(function(doctor) {
    return { // return what new object will look like
        doctorNumber:doctor.number,
        playedBy: doctor.actor,
        
    };
});
1 Like

Assuming that above is javascript, the equivalent in Elixir would be:

doctors=[
  %{ doctorNumber: "#9", playedBy: "Christopher Eccleston", yearsPlayed: 1 },
  %{ doctorNumber: "#10", playedBy: "David Tennant", yearsPlayed: 6 },
  %{ doctorNumber: "#11", playedBy: "Matt Smith", yearsPlayed: 4 },
  %{ doctorNumber: "#12", playedBy: "Peter Capaldi", yearsPlayed: 1 }
]

So getting just a list of maps containing all of the dockerNumber's and playedBy's only would be as a direct translation of your next javascript (assuming you meant to swap the keys and values since they were backwards):

doctors = Enum.map(doctors, fn doctor -> %{
  number: doctor.doctorNumber,
  actor: doctor.playedBy
} end)

As seen here:

iex> doctors=[
  %{ doctorNumber: "#9", playedBy: "Christopher Eccleston", yearsPlayed: 1 },
  %{ doctorNumber: "#10", playedBy: "David Tennant", yearsPlayed: 6 },
  %{ doctorNumber: "#11", playedBy: "Matt Smith", yearsPlayed: 4 },
  %{ doctorNumber: "#12", playedBy: "Peter Capaldi", yearsPlayed: 1 }
]
[%{doctorNumber: "#9", playedBy: "Christopher Eccleston", yearsPlayed: 1},
 %{doctorNumber: "#10", playedBy: "David Tennant", yearsPlayed: 6},
 %{doctorNumber: "#11", playedBy: "Matt Smith", yearsPlayed: 4},
 %{doctorNumber: "#12", playedBy: "Peter Capaldi", yearsPlayed: 1}]
iex> doctors = Enum.map(doctors, fn doctor -> %{
  number: doctor.doctorNumber,
  actor: doctor.playedBy
} end)
[%{actor: "Christopher Eccleston", number: "#9"},
 %{actor: "David Tennant", number: "#10"},
 %{actor: "Matt Smith", number: "#11"},
 %{actor: "Peter Capaldi", number: "#12"}]

There are other more traditionally elixir’y ways, but for such a simple example this is a simple output. :slight_smile:

3 Likes

Woah, thanks a lot ,excellent ! can we also club filter with existing map, just curious though !

doctors = doctors.filter(function(doctor) {
    return doctor.begin > 2000; // if truthy then keep item
}).map(function(doctor) {
    return { // return what new object will look like
        doctorNumber: "#" + doctor.number,
        playedBy: doctor.actor,
        yearsPlayed: doctor.end - doctor.begin + 1
    };
});
1 Like

The ‘traditional’ Elixir way of writing that is in one of two forms, either via Enum piping:

doctors
|> Enum.filter(fn doctor -> doctor.begin > 2000 end)
|> Enum.map(fn doctor -> %{
  doctorNumber: "#" <> to_string(doctor.number),
  playedBy: doctor.actor,
  yearsPlayed: doctor.end - doctor.begin + 1
} end)

Or via a for comprehension:

for
  %{doctorNumber: doctorNumber, playedBy: playedBy} <- doctors, # First grab each doctor out of the list (you can also match out parts here too, but it is numerous in this example so keeping it short)
  doctor.begin > 2000, # Then filter on a condition
  do: %{ # Then return the result after filtering, this example can be arbitrarily complex and remains easier to read when it does
    doctorNumber: "#" <> to_string(doctor.number),
    playedBy: doctor.actor,
    yearsPlayed: doctor.end - doctor.begin + 1
  }

Or something like that. :slight_smile:

EDIT: Also, this forum is Discourse, it supports normal markdown, so using code fences like:

```elixir
def Some elixir code
```
```javascript
var Some javascript code
```

Gets turned into this:

def Some elixir code
var Some javascript code

With syntax coloring and all. :slight_smile:

2 Likes

Thank you , sure moving forword will take the markdown factor into account.
and also edited my queries as per markdowns

1 Like

Awesome! The formatting looks perfect!

Please ask if you have any other questions or conversions, I like the challenges. :slight_smile:

3 Likes

Hi ,
I have a query, I would like to make dynamic Sql statements for multiple where clauses in elixir and pass it across to raw queries.
Please note i use sql.format in my nodejs code to prevent “SQL injection”
my existing javascript code looks likes this

var interns = [{
    "name": "donnie"
}, {
    "age": 16
}, {
    "gender": "male"
}]


var selector = ""
interns.forEach(function(item, index) {

    var searchvalue = item[Object.keys(item)];
    var searchkey = Object.keys(item)[0];


    if (selector == "") {
        selector = "and " + (searchkey) + " LIKE " + "'%" + (searchvalue) + "%'";
    } else {
        selector = selector + " and  " + (searchkey) + " LIKE " + "'%" + (searchvalue) + "%'";
    }


})
console.log(selector)

, how can we achieve the same in elixir .
my progress is kinda stuck in how to trigger else clause and append to z variable

user_params=[%{"age" => 19}, %{"name" => "matt213"}, %{"sex" => "male"}]

sqlwhere=1;
z="";
for norm <- user_params do

for {k, x} <- norm do
  IO.inspect  sqlwhere
 if sqlwhere <=1  do
   
   intern=" and #{k} = '#{x}'"
   x = "and #{k}="
 y = "'#{x}'"
 z = x <> " " <> y
  
  IO.inspect  z
  sqlwhere=sqlwhere+1;


else
 IO.inspect  "unable to get here !"
#     j = "#{k}"
# l = "#{x}"
# k = j <> " " <> l
#    IO.inspect k

   #list=list+"#{list} where #{k} = '#{x}'"  
end
end

end
1 Like

Using strings for building SQL is risky. Fortunately with ecto, we don’t have to do it:

user_params = [%{"age" => 19}, %{"name" => "matt213"}, %{"sex" => "male"}]
user_params
|> Enum.map(&Enum.to_list/1)
|> Enum.reduce(Foo, fn [{key, value}], query ->
  field = String.to_existing_atom(key)
  search = "%#{value}%"
  from q in query, where: like(field(q, ^field), ^search)
end

where Foo is the schema you want to search. I would also probably employ some more advanced mechanism for filtering allowed fields than String.to_existing_atom/1, but for an example it’s good enough.

3 Likes

Hi,

Apologise for earlier post of in process codebase and case scenarios this is my updated codebase;

ui = [%{"age" =&gt; 19}, %{"fullname" =&gt; "todd"}]
lists=ui
|&gt; Enum.map(&Enum.to_list/1)
|&gt; Enum.reduce(User, fn [{key, value}], query -&gt;
  field = String.to_existing_atom(key)
  search = "%#{value}%"
  Repo.all from(q in query, where: like(field(q, ^field), ^search), select: {q.fullname, q.age})
end) 
IO.inspect  lists

However , I am getting below exception for the same . could you please provide any pointers on same

** (exit) an exception was raised:
    ** (Protocol.UndefinedError) protocol Ecto.Queryable not implemented for [{"
Todd", 19}]
        (ecto) lib/ecto/queryable.ex:1: Ecto.Queryable.impl_for!/1
        (ecto) lib/ecto/queryable.ex:9: Ecto.Queryable.to_query/1
1 Like

You don’t want to do Repo.all inside the Enum.reduce callback. The idea is that reduce is reducing over a single query, building it up with each iteration. The return value of the Enum.reduce is the query that you want to execute.

user_params = [%{"age" => 19}, %{"name" => "matt213"}, %{"sex" => "male"}]
query =
  user_params
  |> Enum.map(&Enum.to_list/1)
  |> Enum.reduce(Foo, fn [{key, value}], query ->
    field = String.to_existing_atom(key)
    search = "%#{value}%"
    from q in query, where: like(field(q, ^field), ^search)
  end

users = Repo.all(query)

Or simply:

users =
  user_params
  |> Enum.map(&Enum.to_list/1)
  |> Enum.reduce(Foo, fn [{key, value}], query ->
    field = String.to_existing_atom(key)
    search = "%#{value}%"
    from q in query, where: like(field(q, ^field), ^search)
  end
  |> Repo.all
3 Likes

Affirmative , now it seems quite clear the functional usage of repo, thanks !
one more query what modification is required in code if the incoming user_params look like this

user_params = %{"age" => 19, "fullname" => "matt213", "sex" => "male"}

1 Like

This is a simple enough change I think it’s best if you gave it a shot on your own. If you have something like

user_params = %{"age" => 19, "fullname" => "matt213", "sex" => "male"}

what does doing

user_params |> Enum.map(fn thing -> IO.inspect(thing) end)

print?

What does this tell you about the shape of each item when you map over a map, and how might that fit into your Enum.reduce call?

1 Like

hi ,
below is my modified codebase , it throws below error

** (exit) an exception was raised:
** (Protocol.UndefinedError) protocol Ecto.Queryable not implemented for
[#Ecto.Query from u in HelloPhoenix.User, where: like(u.age, ^"%19%")>, #Ecto.Query
from u in HelloPhoenix.User, where: like(u.name, ^"%matt213%")>]

Modified Codebase

      query=user_params
      |> Enum.map(&Enum.to_list/1)
      |> Enum.reduce(User, fn thing, query ->

 for {key, value} <- thing do

    field = String.to_existing_atom(key)
    search = "%#{value}%"
    from q in query, where: like(field(q, ^field), ^search)

end

  end)
 user = Repo.all(query)

IO.inspect user
1 Like