brainlid

brainlid

Podcast: ThinkingElixir 129 - Time series data with Timescale DB

Episode 129 of Thinking Elixir. Tracking, analyzing and visualizing time series data can add a lot of business value to a project! We met up with Dave Lucia to learn more about Timescale DB, a PostgreSQL extension that adds time series tools to our regular database. Dave also created a timescale hex package to make it easier to work with hypertables and hyperfunctions. We learn why Timescale DB makes sense over other options, how to get started with it, example use cases, helpful resources and more!

First 6 of 6 Posts! Switch mode

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

We’ve been using TimescaleDB at CargoSense for a couple of years now, and it has worked really well for us. In particular, the compression feature gives us 96%+ compression rate which makes a huge difference. Concretely we have a ~terrabyte’s worth of data taking up 55 gigabytes and for our query patterns, we see negligible performance performance impact for queries on compressed chunks.

For us it’s a sweet spot on trade offs. We get to keep the simplicity of running “just” postgres, we get to keep all our transactional guarantees with other code, and we also get to store bunch of timeseries data.

The only “gotcha” is that, like all partitioned tables, you need to be storing “relevant time range” values around so that you can query the right range. Eg if you had a sensor and you wanted to query its temperature readings in a shipment, you’d want to do a query like

Repo.all(from r in Reading,
  where: r.sensor_id == ^sensor.id,
  where: r.recorded_at >= ^shipment.starts_at and r.recorded_at <= ^shipment.ends_at

The r.recorded_at >= ^shipment.starts_at and r.recorded_at <= ^shipment.ends_at bit is important, even if a sensor is never used in multiple shipments because you need to help the query planner know in which chunks to go looking for readings.

brainlid

brainlid

Hi @benwilson512!

I’m glad to hear you’ve had such a good experience with it as well. And thanks for the query tip. Very helpful!

thbar

thbar

Hello @benwilson512!

We’re considering using TimescaleDB for long term metrics.

How are you integrating it with Elixir? Are you using a library (e.g. GitHub - davydog187/timescale: TimescaleDB made easy with Ecto · GitHub) or something more direct?

Thanks!

Schultzer

Schultzer

You might not need timescale with Postgres today.

Designing high-performance time series data tables on Amazon RDS for PostgreSQL | AWS Database Blog

dimitarvp

dimitarvp

This is not specific to RDS, correct? Seems like it refers to partitioning tables in a vanilla Postgres.

Schultzer

Schultzer

Yes, it’s Postgres in general!

Where Next?

Trending in Podcasts Top

PJUllrich
Announcing Macro Mayhem @gus and I have started a new podcast called Macro Mayhem! We cover Elixir news and general industry topics ever...
New
ancatrusca
Sharing a recent BEAM There, Done That episode with Bart Blast, creator of Hologram. Worth a read for anyone who’s thought seriously abou...
New
ancatrusca
Sharing the latest BEAM There, Done That episode, which covers Zig and Zigler in more practical depth than I’ve seen in one place before....
New
ancatrusca
Sharing the latest BEAM There, Done That episode. This one is genuinely historic - the people who built the JAM and early BEAM, telling t...
New

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge &amp; Solve. They are GUI (Emerge) and State management (S...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
type1fool
I just stumbled on a newly redesigned elixir-lang.org. :tada: It looks like @Software_Mansion did the work, and I think it is generally a...
New
akoutmos
@hugobarauna and I (Alex Koutmos) have been hard at work on writing a book on Nerves that takes you from simply blinking LEDs to building...
New

We're in Beta

About us Mission Statement