Storing any type of Elixir data in PostgreSQL

Phoenix Forum Questions / Help
, , ,
1

I stumbled upon this:

https://medium.com/@kaisersly/storing-any-elixir-data-in-postgres-e8f93367f473

I am looking for a way to store transient data in PostgreSQL. Basically, LLMChains that can grow big in size.

Is there any better way to do it ? The article suggests using :binary and Term. Or is this actually the perfect way to do it ?

2

How big is big? Past a certain point you probably want an object store. Before that though binary and term are quite useful. If you want want to be able to search it from within the DB though consider :jsonb

2 Likes
3

If size is an issue you either use term_to_binary with compression or just use a full-blown object store as @benwilson512 suggested.

2 Likes
4

@benwilson512 What do you mean by object store?

5

Do you need to index them?

1 Like
6

Amazon S3 or Minio are examples of object stores. They’re also often called blob storage.

7

I need to look them up by a key, so yes. The data can get big, since it is a chain of AI messages

8

Maybe, just maybe, you can add two columns, one is for the key (indexed), and the other is for the arbitrary Erlang/Elixir data structure encoded in binary.

1 Like
9

I think it all comes down to how big the data for each item will be. Yes, if it gets big, then maybe yes, this way in a standard PostgreSQL DB, not memory, mnesia or ETS.

10

Can it be stored compressed? The binary data?

11

External Term Format — erts v15.1.3 seems to say that the data is zlib-compressed.

2 Likes
12

:erlang.term_to_binary(term) compresses the data by default.
:erlang.term_to_binary(term, compressed: 9) compress the data at highest level.
:erlang.term_to_binary(term, compressed: false) turns off the compression.

4 Likes