sabri

sabri

Which database for time-series data?

Hi,

I am building my channels based app, users will be sending their lat, long as messages each 1 second, I need to store all the messages, there could be hundred of thousands users sending their locations, the message size is less than 100 byte.

The question is, what is the recommended db to go with? As I will be storing all those messages in real time.. ? Each process (channel) will store the message once it’s received via Task.async

Please advice!

Most Liked Responses

uranther

uranther

InfluxDB came to mind because it focuses on storing time-series data. There is also TrailDB which was recently released by AdRoll but may not fit your use case as well.

In general, I would think about appending these messages to a log like Apache Kafka or RabbitMQ, and then creating subscribers/consumers which store that data in special purpose databases like InfluxDB. This allows for more flexibility as your application grows. Consider the following diagrams:



sources:

Qqwy

Qqwy

TypeCheck Core Team

Are you going to need all that information forever? I think probably not (Rather, you’d want to create some smoothing algorithm that only stores a location once it is far enough from a users previous location).

If I am correct in that assumption, I think that you could store the first, real-time information using Mnesia (which is Erlangs built-in in-memory database that is made for concurrent read/writes), and after processing this information, store the smoothed values in a more conventional relational database such as PostgreSQL.

mkunikow

mkunikow

Yes I agree Apache Kafka is the best message queue right now. It can very easy scale adding more nodes if needed.

But the main question is what you will do with this data, how do you process this data and how do you want to access this data later.

If you have kafka you can have many consumers, process data and put it to different destinations.

There was interesting article on Arts Technica Power tools: Sorting through the crowded specialized database toolbox

You could put also data into Welcome to Apache Solr - Apache Solr or Elasticsearch: The Official Distributed Search & Analytics Engine | Elastic if you want to have data optimized for search.

if you are only interested to query data over time maybe time maybe influx db is good for you
Overview of influxdb

Summary:
You always optimize data how you access it later.

Where Next?

Popular in Questions Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
JulienCorb
I am trying to implement my new.html.eex file to create new posts on my website. new.html.eex: <h1>Create Post</h1> <%= ...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> somethi...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New

Other popular topics Top

albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement