brainlid

brainlid

Podcast: ThinkingElixir 129 - Time series data with Timescale DB

Episode 129 of Thinking Elixir. Tracking, analyzing and visualizing time series data can add a lot of business value to a project! We met up with Dave Lucia to learn more about Timescale DB, a PostgreSQL extension that adds time series tools to our regular database. Dave also created a timescale hex package to make it easier to work with hypertables and hyperfunctions. We learn why Timescale DB makes sense over other options, how to get started with it, example use cases, helpful resources and more!

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

We’ve been using TimescaleDB at CargoSense for a couple of years now, and it has worked really well for us. In particular, the compression feature gives us 96%+ compression rate which makes a huge difference. Concretely we have a ~terrabyte’s worth of data taking up 55 gigabytes and for our query patterns, we see negligible performance performance impact for queries on compressed chunks.

For us it’s a sweet spot on trade offs. We get to keep the simplicity of running “just” postgres, we get to keep all our transactional guarantees with other code, and we also get to store bunch of timeseries data.

The only “gotcha” is that, like all partitioned tables, you need to be storing “relevant time range” values around so that you can query the right range. Eg if you had a sensor and you wanted to query its temperature readings in a shipment, you’d want to do a query like

Repo.all(from r in Reading,
  where: r.sensor_id == ^sensor.id,
  where: r.recorded_at >= ^shipment.starts_at and r.recorded_at <= ^shipment.ends_at

The r.recorded_at >= ^shipment.starts_at and r.recorded_at <= ^shipment.ends_at bit is important, even if a sensor is never used in multiple shipments because you need to help the query planner know in which chunks to go looking for readings.

brainlid

brainlid

Hi @benwilson512!

I’m glad to hear you’ve had such a good experience with it as well. And thanks for the query tip. Very helpful!

Schultzer

Schultzer

You might not need timescale with Postgres today.

Designing high-performance time series data tables on Amazon RDS for PostgreSQL | AWS Database Blog

Where Next?

Popular in Podcasts Top

wolf4earth
The Elixir Mix panel takes the helm to talk about helping onboard and transition new developers onto an Elixir team. They discuss helping...
New
brainlid
In episode 54 of Thinking Elixir, Lucas San Román explains his library Sourceror and how it was created to solve some AST parsing limitat...
New
brainlid
In episode 91 of Thinking Elixir, José Valim returns to continue with part 3 of our 5 part series as we count down to the 10 year anniver...
New
brainlid
Episode 119 of Thinking Elixir. Kat Marchán was the maintainer of NPM CLI for 5 years but now works at Microsoft on the VisualStudio prod...
New
brainlid
In episode 97 of Thinking Elixir, Dave Lucia released v2.0 of AvroEx, an Elixir library for working with Avro protocols and schemas. To u...
New
brainlid
Day 1 and 2 of Livebook’s first launch week announcements, a new JsonSchema compiler named Exonerate, Ecto 3.10.0 brings changeset improv...
New
brainlid
In episode 101 of Thinking Elixir, Ben Johnson explains his project Litestream.io, an OpenSource tool that replicates SQLite databases to...
New
brainlid
In episode 86 of Thinking Elixir, we talk with Derrick Reimer about his product SavvyCal. He used Elixir as a solo dev to create a servic...
New
brainlid
Some people excitedly talk about the “end of localhost development” when developer machines move to the cloud. Presumably this is better ...
New
brainlid
In episode 109 of Thinking Elixir, we talk about a new library in the Nx ecosystem under active development called NxSignal by Paulo Vale...
New

Other popular topics Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31142 143
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

We're in Beta

About us Mission Statement