sabri

sabri

Hi,

I am building my channels based app, users will be sending their lat, long as messages each 1 second, I need to store all the messages, there could be hundred of thousands users sending their locations, the message size is less than 100 byte.

The question is, what is the recommended db to go with? As I will be storing all those messages in real time.. ? Each process (channel) will store the message once it’s received via Task.async

Please advice!

Most Liked

uranther

uranther

InfluxDB came to mind because it focuses on storing time-series data. There is also TrailDB which was recently released by AdRoll but may not fit your use case as well.

In general, I would think about appending these messages to a log like Apache Kafka or RabbitMQ, and then creating subscribers/consumers which store that data in special purpose databases like InfluxDB. This allows for more flexibility as your application grows. Consider the following diagrams:



sources:

Qqwy

Qqwy

TypeCheck Core Team

Are you going to need all that information forever? I think probably not (Rather, you’d want to create some smoothing algorithm that only stores a location once it is far enough from a users previous location).

If I am correct in that assumption, I think that you could store the first, real-time information using Mnesia (which is Erlangs built-in in-memory database that is made for concurrent read/writes), and after processing this information, store the smoothed values in a more conventional relational database such as PostgreSQL.

mkunikow

mkunikow

Yes I agree Apache Kafka is the best message queue right now. It can very easy scale adding more nodes if needed.

But the main question is what you will do with this data, how do you process this data and how do you want to access this data later.

If you have kafka you can have many consumers, process data and put it to different destinations.

There was interesting article on Arts Technica Power tools: Sorting through the crowded specialized database toolbox

You could put also data into Welcome to Apache Solr - Apache Solr or Elasticsearch: The Official Distributed Search & Analytics Engine | Elastic if you want to have data optimized for search.

if you are only interested to query data over time maybe time maybe influx db is good for you
Overview of influxdb

Summary:
You always optimize data how you access it later.

Last Post!

gausby

gausby

No problem. We are all here to learn.

Perhaps Basho should have rebranded Riak TS as something different than “Riak.” They are based on the same technology, but you interface them differently and they are tuned to do different things.

Where Next? Top

Trending in Questions Top

RSP87
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
kpanic
Hi everyone, I am toying with the idea of building a “match maker” for giving personal help to people that wants to start coding. I sta...
New
nseaSeb
Hello, I know there is an approach for handling lists that allows for optimized traversal, but I can’t recall the specific method (somet...
New
brecabral
Documentation While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
velrest
So my question is quite simple and i have found no conclusive answer on forum, google or AI. Should we use :erlang.float for Integer to ...
New
asweet-confluent
I recently noticed that Elixir’s Logger defaults its primary log level to :debug when no :logger, :level application configuration is pre...
New
ryanwinchester
apply_graft/2 doesn’t rewrite an add_many sub-workflow’s deps on an add step. Grafted jobs cancel with “upstream job was deleted” Version...
New

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
mhanberg
Hi everyone! The first release candidate for the Expert language server project is now available! We’ve published a press release detai...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
webofbits
With AI doing more of the implementation work, I’ve been wondering how much coding I should deliberately keep doing myself. My main conc...
#ai
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews