Missing something with Stream to Enum

ddombrow

Missing something when going through the Little Elixir book. Made my own function to grab a csv and make weather requests.

The result is the result of the first operation (a correct temperature response from a service, followed by the cities list. I expected a list of outputs only. Any pointers?

def get_cities_weather(csv_path) do
  alias NimbleCSV.RFC4180, as: CSV

  cities = csv_path
  |> File.stream!
  |> CSV.parse_stream
  |> Stream.map(fn [city, city_ascii, lat, lng, pop, country, iso2, iso3, province] ->
      %{name: city_ascii, lat: lat, lng: lng}
    end)

  cities
  |> Enum.map_every(500, fn city ->
    Metex.Worker.temperature_of(city)
  end)

end

Produces:

[{:ok, 40.8}, %{lat: "34.5167011", lng: "65.25000063", name: "Chaghcharan"},
 %{lat: "31.58299802", lng: "64.35999955", name: "Lashkar Gah"},
 %{lat: "31.11200108", lng: "61.88699752", name: "Zaranj"},
 %{lat: "32.63329815", lng: "65.86669865", name: "Tarin Kowt"},
 %{lat: "32.85000016", lng: "68.41670453", name: "Zareh Sharan"}...]

8 comments

#streams #troubleshooting

1 4081 8

2017-03-30 14:49:16 UTC

Showing Posts 1 to 8

Show Best Posts
Show All (oldest first)
Show All (newest first)

NobbZ

map_every(enumerable, nth, fun)

map_every(t, non_neg_integer, (element -> any)) :: list

Returns a list of results of invoking fun on every nth item of enumerable, starting with the first element.

So you have to wait another 495 elements after the ellipsis, before you will get a processed item again.

Perhaps you want plain old Enum.map/2 or Stream.map/2 instead?

Post #1

PatNowak

I don’t know what exactly you want to achieve, but there’s Enum.into if you want to convert these maps into lists and - most of all - there’s Enum.to_list for converting stream into list.

Post #2

ddombrow

OH! It’s just returning the original input if it skips it… I was not thinking of that. In my head it was going to even return every 500th item… Thanks for pointing it out.

Post #3

NobbZ

If you want that behaviour, you probably want Enum.take_every/2 or Stream.take_every/2.

It could look like this then:

def get_cities_weather(csv_path) do
  alias NimbleCSV.RFC4180, as: CSV

  csv_path
  |> File.stream!
  |> CSV.parse_stream
  |> Stream.map(fn [city, city_ascii, lat, lng, pop, country, iso2, iso3, province] ->
      %{name: city_ascii, lat: lat, lng: lng}
    end)
  |> Stream.take_every(500)
  |> Enum.map(fn city ->
    Metex.Worker.temperature_of(city)
  end)
end

Post #4

ddombrow

take_every was what I wanted, thanks!

Question. If I did Stream.take_every it’s lazier right? It won’t actually do anything until I Enum.map in the next step?

Post #5

NobbZ

Yupp. A stream does nothing until you either Stream.run/1 it or use it as an Enumerable.

Post #6

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

or use it as an Enumerable

Minor nitpick: Enumerables are neither lazy nor eager. Stream deals with Enumerables lazily, and Enum deals with them eagerly.

So a stream does nothing until it is consumed. It can be consumed by Stream.run or any of the Enum functions that eagerly consume their inputs.

Post #7

NobbZ

But an important one! I thought if enumerable is the right word here quite a moment. Names are hard especially if you have to deal an remember the relations between similar names.

Post #8

— All posts loaded —