idi527
Data structure for a time series
How would one store a time series (updates in “real time”) with elixir/erlang? I’m getting some market data over a socket and would like to know how would you guys decide to store it efficiently. Ideally, I would also like to run some rolling calculations (not sure if it’s how it called, I want to apply some aggregation over some pre-scpecified time period (like for the last minute)) on the incoming data.
It’s not a real project, it’s just for a hobby. So I would like to just use erlang/elixir tools, and not reach out for leveldb or anything like that.
Found this discussion on using mnesia for storing time series data Mnesia not suitable for time series storage?, but even mnesia seems like overkill to me.
First Post!
idi527
I think I just need to keep a list of a fixed length in process’s memory. So that when the new data comes in, that last element gets removed from the list and the new data gets put in the head of the list.
old_list = [b, c, d]
# new data `a` comes in
new_list = update(old_list, a)
new_list == [a, b, c]
Don’t know how to do that without pointers … Will look into how :queue module works.
defmodule Quote.History do
@type t :: {new :: list, old :: list}
def new do
{[], []}
end
@doc """
iex> new([3, 2, 1])
{[3], [1, 2]}
"""
@spec new(list) :: t
def new(list)
def new([h | t]) do
{[h], :lists.reverse(t)}
end
@doc """
iex> history = new([3, 2, 1])
iex> history |> push(4) |> to_list()
[4, 3, 2]
"""
@spec to_list(t) :: list
def to_list({new, old}) do
new ++ :list.reverse(old)
end
@doc """
iex> history = {[3], [1, 2]} = new([3, 2, 1])
iex> push(history, 4)
{[4, 3], [2]}
iex> push({[4, 3], [2]}, 5)
{[5, 4, 3], []}
iex> push({[5, 4, 3], []}, 6)
{[6, 5], [4]}
"""
@spec push(t, any) :: t
def push(hisroty, element)
def push({new, [_ | rest]}, element) do
{[element | new], rest}
end
def push({[new | rest], []}, element) do
[_ | old] = :lists.reverse(rest)
{[element, new], old}
end
def push({[], []}, element) do
{[], [element]}
end
end
Something like this maybe. Don’t like all these calls to :lists.reverse …
Seems to work. But also seems hacky …
Interactive Elixir (1.6.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> history = Quote.History.new [3, 2, 1]
{[3], [1, 2]}
iex(2)> Quote.History.to_list history
[3, 2, 1]
iex(3)> Quote.History.push history, 4
{[4, 3], [2]}
iex(4)> history = Quote.History.push history, 4
{[4, 3], [2]}
iex(5)> Quote.History.to_list history
[4, 3, 2]
iex(6)> history = Quote.History.push history, 5
{[5, 4, 3], []}
iex(7)> Quote.History.to_list history
[5, 4, 3]
iex(8)> history = Quote.History.push history, 6
{[6, 5], [4]}
iex(9)> Quote.History.to_list history
[6, 5, 4]
Most Liked
Qqwy
A couple of months ago, I was working on something that exactly keeps track of things like market data, where you have a sliding window of datapoints that you want to aggregate in possibly multiple ways.
The code was uploaded to Github; it’s not yet on Hexpm because it is still a little bare-bones (although there are tests!).
Actually, the tests might be the best explanation as to how it works
.
kip
When I researching into this topic a while back I thought to use ETS ordered sets. Basically use the ordered set with a counter like {n, payload} and monotonically increase n. Then you can use :ets.next/1 for access. But of course then you need a sweeper to delete the head of the set on some kind of regular basis. Or manage n predefined slots in an ordered set and manage the read and write “pointers” manually.
I also tried the list-based approach you describe above but it does involve a lot of list copying on pretty much each update. I never finished up my experiments so I have no useful data to offer.
mbuhot
Can you amortize the cost of popping the old items from the list by allowing the list to grow up to 2x the desired size, then calling Enum.take to keep the required prefix?
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex









