9mm
How to quickly add to list in map?
Hello,
I have a collection like:
%{stats_a: [4, 5, 6], stats_b: [1, 2, 3]} # not actually integers (see next question)
Each of the 2 lists will be ~1 million items long.
I have a method which needs to push an item to beginning of each list. Ideally, it would in real-time chop the oldest/last items of the list so that it’s never longer than 1,000,000 items exactly
If theres no super fast way to do this however I will set a timer to do it every 1000ms… if there is an O(n) fast way to do it though I’ll do it in realtime. I need this timer anyway to aggregate stats so I’m not bothered if it needs to go there.
I will be pushing to it ~300-500 times/second.
Here’s what I had so far (without the limiting part yet…)
Is this crazy what I wrote?
def handle_cast({:push_stats, key, stats}, state) do
items = [stats | state[key]]
new_state = state |> Map.put(key, items)
{:noreply, new_state}
end
Most Liked
peerreynders
defmodule Demo do
def merge_values(_key, value1, value2),
do: value1 + value2
def reducer({key, data}, aggregate) do
merge_data = fn all_data -> Map.merge(all_data, data, &merge_values/3) end
Map.update(aggregate, key, data, merge_data)
end
def run(list),
do: List.foldl(list, %{}, &reducer/2)
end
list = [
{:group_1, %{timeout: 1, failure: 0, hits: 0}},
{:group_1, %{timeout: 0, failure: 0, hits: 0}},
{:group_2, %{timeout: 1, failure: 1, hits: 0}}
]
IO.inspect(Demo.run(list))
$ elixir demo.exs
%{
group_1: %{failure: 0, hits: 0, timeout: 1},
group_2: %{failure: 1, hits: 0, timeout: 1}
}
michalmuskala
One thing to consider might be using an ets table instead of a server with state. In general, if you need high performance storage, ets tables are a very good solution. You could still have a process periodically reading from the table and doing some statistics.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









