Enum.uniq_by Counter

[kitplummer]

Been a bit since I’ve used Elixir. This is probably simple so apologies for looking for an easy answer.

Say I have a map, and want to get the unique items, but also count the number of duplicates removed.

d = Enum.uniq_by([{1, :x}, {2, :y}, {1, :z}], fn {x, _} ->
  count = count + 1
  IO.inspect(count, label: "C")
  x
end)

I know this currently doesn’t work.

How would I go about getting the value of count out of the function along with the new map?

[kitplummer]

Maybe this is a dupe…

Getting duplicates instead of unique values

I know that uniq/1 & uniq_by/2 return a list of unique values, but what is the best way to get a list of duplicate values? I noticed that the code for uniq & uniq_by could easily collect a list of duplicates while unique values are collected; so I copied this code and modified it to return a tuple with a list of unique values and a list of duplicates. But have I missed some existing functionality to do this?

[kokolegorille]

You can get the count by substracting length of result from length of d.

[kitplummer]

Ha. That makes too much sense.

[kitplummer]

Coming back to this for posterity. After a bit more thought I’m still stuck. I’m processing a very large JSON file as input, as a stream, writing the output of a series of steps to a CSV. Part of this process is the dedupe by the Enum.uniq_by(). Since I’m never really holding the original map, and the new map I can’t do the easy math.

[LostKobrakai]

Since elixir 1.10 there is Enum.frequencies_by, which should mostly do what you need.

[kitplummer]

Thanks, that’s great!

I’ve had to move around the in-stream single counting - and use an Agent to capture instances of various conditions during the stream.