acrolink
I stumbled upon this:
https://medium.com/@kaisersly/storing-any-elixir-data-in-postgres-e8f93367f473
I am looking for a way to store transient data in PostgreSQL. Basically, LLMChains that can grow big in size.
Is there any better way to do it ? The article suggests using :binary and Term. Or is this actually the perfect way to do it ?
Where Next?
Trending in Questions
Other Trending Topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Bringing learning, work and compliance together since 1987.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Bringing learning, work and compliance together since 1987.
**Building the future of software with AI, trust, and human connection.**
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
benwilson512
How big is big? Past a certain point you probably want an object store. Before that though binary and term are quite useful. If you want want to be able to search it from within the DB though consider :jsonb
dimitarvp
If size is an issue you either use
term_to_binarywith compression or just use a full-blown object store as @benwilson512 suggested.acrolink
@benwilson512 What do you mean by object store?
Aetherus
Do you need to index them?
hkrutzer
Amazon S3 or Minio are examples of object stores. They’re also often called blob storage.
acrolink
I need to look them up by a key, so yes. The data can get big, since it is a chain of AI messages
Aetherus
Maybe, just maybe, you can add two columns, one is for the key (indexed), and the other is for the arbitrary Erlang/Elixir data structure encoded in binary.
acrolink
I think it all comes down to how big the data for each item will be. Yes, if it gets big, then maybe yes, this way in a standard PostgreSQL DB, not memory, mnesia or ETS.
acrolink
Can it be stored compressed? The binary data?
dimitarvp
External Term Format — OTP 29.0.2 (erts 17.0.2) seems to say that the data is zlib-compressed.