9mm

9mm

How to quickly add to list in map?

Hello,

I have a collection like:

%{stats_a: [4, 5, 6], stats_b: [1, 2, 3]} # not actually integers (see next question)

Each of the 2 lists will be ~1 million items long.

I have a method which needs to push an item to beginning of each list. Ideally, it would in real-time chop the oldest/last items of the list so that it’s never longer than 1,000,000 items exactly

If theres no super fast way to do this however I will set a timer to do it every 1000ms… if there is an O(n) fast way to do it though I’ll do it in realtime. I need this timer anyway to aggregate stats so I’m not bothered if it needs to go there.

I will be pushing to it ~300-500 times/second.

Here’s what I had so far (without the limiting part yet…)

Is this crazy what I wrote?

  def handle_cast({:push_stats, key, stats}, state) do
    items = [stats | state[key]]
    new_state = state |> Map.put(key, items)
    {:noreply, new_state}
  end

Most Liked

peerreynders

peerreynders

defmodule Demo do
  def merge_values(_key, value1, value2),
    do: value1 + value2

  def reducer({key, data}, aggregate) do
    merge_data = fn all_data -> Map.merge(all_data, data, &merge_values/3) end
    Map.update(aggregate, key, data, merge_data)
  end

  def run(list),
    do: List.foldl(list, %{}, &reducer/2)
end

list = [
  {:group_1, %{timeout: 1, failure: 0, hits: 0}},
  {:group_1, %{timeout: 0, failure: 0, hits: 0}},
  {:group_2, %{timeout: 1, failure: 1, hits: 0}}
]

IO.inspect(Demo.run(list))
$ elixir demo.exs
%{
  group_1: %{failure: 0, hits: 0, timeout: 1},
  group_2: %{failure: 1, hits: 0, timeout: 1}
}
michalmuskala

michalmuskala

One thing to consider might be using an ets table instead of a server with state. In general, if you need high performance storage, ets tables are a very good solution. You could still have a process periodically reading from the table and doing some statistics.

peerreynders

peerreynders

There is Erlang’s queue data type.

Where Next?

Popular in Questions Top

Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New

Other popular topics Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36128 110
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 47930 226
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement