acrolink
Storing any type of Elixir data in PostgreSQL
I stumbled upon this:
https://medium.com/@kaisersly/storing-any-elixir-data-in-postgres-e8f93367f473
I am looking for a way to store transient data in PostgreSQL. Basically, LLMChains that can grow big in size.
Is there any better way to do it ? The article suggests using :binary and Term. Or is this actually the perfect way to do it ?
Trending in Questions
Other Trending Topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #performance
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Bringing learning, work and compliance together since 1987.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 10 of 11 Posts
benwilson512
How big is big? Past a certain point you probably want an object store. Before that though binary and term are quite useful. If you want want to be able to search it from within the DB though consider :jsonb
dimitarvp
If size is an issue you either use
term_to_binarywith compression or just use a full-blown object store as @benwilson512 suggested.acrolink
@benwilson512 What do you mean by object store?
Aetherus
Do you need to index them?
hkrutzer
Amazon S3 or Minio are examples of object stores. They’re also often called blob storage.
acrolink
I need to look them up by a key, so yes. The data can get big, since it is a chain of AI messages
Aetherus
Maybe, just maybe, you can add two columns, one is for the key (indexed), and the other is for the arbitrary Erlang/Elixir data structure encoded in binary.
acrolink
I think it all comes down to how big the data for each item will be. Yes, if it gets big, then maybe yes, this way in a standard PostgreSQL DB, not memory, mnesia or ETS.
acrolink
Can it be stored compressed? The binary data?
dimitarvp
External Term Format — OTP 29.0.2 (erts 17.0.2) seems to say that the data is zlib-compressed.