Which database for time-series data?

Are you going to need all that information forever? I think probably not (Rather, you’d want to create some smoothing algorithm that only stores a location once it is far enough from a users previous location).

If I am correct in that assumption, I think that you could store the first, real-time information using Mnesia (which is Erlangs built-in in-memory database that is made for concurrent read/writes), and after processing this information, store the smoothed values in a more conventional relational database such as PostgreSQL.

InfluxDB came to mind because it focuses on storing time-series data. There is also TrailDB which was recently released by AdRoll but may not fit your use case as well.

In general, I would think about appending these messages to a log like Apache Kafka or RabbitMQ, and then creating subscribers/consumers which store that data in special purpose databases like InfluxDB. This allows for more flexibility as your application grows. Consider the following diagrams:

Pasted image660×495 190 KB

Pasted image660×495 155 KB

Pasted image709×244 25.7 KB

Pasted image704×295 15.8 KB

sources:

• Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)
• The Log: What every software engineer should know about real-time data’s unifying abstraction

Yes I agree Apache Kafka is the best message queue right now. It can very easy scale adding more nodes if needed.

But the main question is what you will do with this data, how do you process this data and how do you want to access this data later.

If you have kafka you can have many consumers, process data and put it to different destinations.

There was interesting article on Arts Technica Power tools: Sorting through the crowded specialized database toolbox

You could put also data into Welcome to Apache Solr - Apache Solr or Elasticsearch: The Official Distributed Search & Analytics Engine | Elastic if you want to have data optimized for search.

if you are only interested to query data over time maybe time maybe influx db is good for you
Overview of influxdb

Summary:
You always optimize data how you access it later.

You might want to try Riak TS.

Riak TS is engineered to be faster than Cassandra.

Thanks, I just discovered Cassandra few days, and now Riak TS!
How would I consume Riak TS inside my elixir app? it seems that Cassandra’s community is much bigger than Riak TS..

Riak TS is kind of new, that’s why the community is rather small at the moment.

You can use the riak client (timeseries release): Releases · basho/riak-erlang-client · GitHub

The official documentation has snippets to work around with querying and data processing.

I have installed Riak via docker:

And I have installed Elixir Riak client:
https://github.com/drewkerrigan/riak-elixir-client

Connecting worked fine:
{:ok, pid} = Riak.Connection.start_link('192.168.99.100', 32774)

However, putting a user has failed, here is my code:

def store_to_riak do
  {:ok, pid} = Riak.Connection.start_link('192.168.99.100', 32774)
  IO.inspect pid
  o = Riak.Object.create(bucket: "user", key: "my_key", data: "Han Solo")
  IO.inspect o
  Riak.put(pid, o)
end

The result:

iex(1)> App.LocationController.store_to_riak
#PID<0.314.0>
%Riak.Object{bucket: "user", content_type: 'application/json', data: "Han Solo",
 key: "my_key",
 metadata: {:dict, 1, 16, 16, 8, 80, 48,
  {[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []},
  {{[], [], [], [], [], [], [], [], [], [],
    [["content-type", 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47,
      106, 115, 111, 110]], [], [], [], [], []}}}, type: :undefined,
 vclock: :undefined}
nil
** (EXIT from #PID<0.312.0>) :disconnected

Interactive Elixir (1.2.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> [error] GenServer #PID<0.314.0> terminating
** (stop) :disconnected
Last message: {:tcp_closed, #Port<0.12186>}
State: {:state, '192.168.99.100', 32774, false, false, :undefined, false, :gen_tcp, :undefined, {[], []}, 1, [], :infinity, :undefined, :undefined, :undefined, :undefined, [], 100}

But, sense I am new to Riak, I don’t know how to handle such errors, if you can help that would be great

Why picking timeseries release?

I suggest using this client and branch for Riak TS.

Isn’t Riak TS the same that will be installed here: Riak Quick Start with Docker – Riak ?