Logging too many log request

In that case Logger isn’t something you look for. You need event store instead of logging platform. Both are in fact logs, but for different purposes.

4 Likes

That’s exactly what Event Sourcing does, it stores events to be able to replay them later when needed., and that’s why @slashdotdash build the EventStore. So, in part you are doing kind of Event Sourcing without knowing you are doing it:

Check the RealWorld example of CQRS by @slashdotdash in Elixir:

Now, if you don’t like to use Postgres as the persitent layer for the events, then the EventStore allows you to provide another storage backend.

You can also read the in progress book from him:

What you don’t want to do is to use any logging library, because they are built for logging, and in logging is ok to loose data.


Maybe this guide can help you:

Or:

https://blog.discord.com/scaling-elixir-f9b8e1e7c29b

The library from Discord to solve distributed sessions:

Kudos for the rich resources provided, I’ve got a lot of reading to do. I will revert with any questions should the need the arise.

It does? But the library specifically states that it uses PostgreSQL, and the guide pages assume it. How do you use another database? Or not even a DB like Kafka?

@kodepett For you to understand better how an event store works I recommend you to read this articles by @alvises :

or

In the articles you will learn that a disk event store uses a key:value append only approach into a file and keeps the index of it in another file, just like Kafka does:

Reading the above articles will give you a very deep understanding how event logs work and how they can be stored, therefore leaving you in a much better position to understand the trade-offs when making the decision how you will gonna proceed in your application.

A simple disk based Event Log library that you can look into the code to understand the principles of key:value with indexes:

A distributed and production ready library for Kafka like events:


Related links in the forum:

3 Likes

See this reply from the author of the library:

Maybe this one:

Vonnegut is a append-only log that follows the file format and API of Kafka 1.0. The server can be run standalone, with 1 or more chains each with 1 or more replicas, or as part of another Erlang release which can talk to it directly.

Did not know about that vonnegut one. Good point.

1 Like

Thanks, but doesn’t that mandate you to use commanded as well? Guess it was a while since I last checked it but for me using a relational DB for an event store is no go. It doesn’t scale well.

As far I am aware you don’t need to use Commanded, but @slashdotdash is the best one to answer you that :wink:

I am from the same opinion and that’s why I am looking to alternatives, because I need a safe backup so that I can recover from a Mnesia disaster:

Probably I will replace the default Mnesia with:

Despite being looking at the safer Mnevis as alternative for Mnesia I still want to use a proper event store to be on the safe side… just in case :wink:

1 Like

After being in a number of such projects I’ll just opt for Kafka or RabbitMQ. It’s too much pain trying to use an RDBMS as an append-only log with data expiration.

Kafka is amazing but the deployment story is quite bad – you need a Zookeeper most of the time, too. So either use a heavily managed Kafka hosting or suffer through putting Kafka in Docker/Kubernetes and never do it again. :003:

2 Likes

In this series of articles we will see the different concepts behind a key-values store engine, implementing a simple engine in Elixir.

Excellent stuff, I’m done going through and implementing the solution. I need to read a bit on the binary aspect under part 2 which is what I’m currently doing. I will update you as and when I’m done with the resources shared and my general opinion based on the knowledge acquired.

Can you share a resource that delves into Elixir Binaries/Bitstrings?
I’m perusing one resource, I need clarification on storing integers/floats as binaries e.g.

<<1, 2, 3>>

Is the above equivalent to the 123 or `“123”, a quick check in iex returns false. What will the above represent. Sorry I’ve to ask a lot of question.

You don’t need to learn binary to save to a log file, you just use :erlang.term_to_binary and then pass it into IO.binwrite(file_descriptor, binary_data) to save into the file and when reading from the file you jsut use :erlang.binary_to_term to get your data decoded:

defmodule FileIO do

  @moduledoc """
  iex(3)> fd = FileIO.open! "test.bin"
  #PID<0.200.0>
  iex(4)> FileIO.append! fd, %{key: "value"}
  %{end_position: 22, file_descriptor: #PID<0.200.0>, size: 22, start_position: 0}
  iex(5)> FileIO.read! "test.bin"
  %{key: "value"}
  """

  @write_mode [:append, :binary]
  # @write_mode [:append, :binary, :delayed_write]
  # @write_mode [:append, :binary, {:delayed_write, 1, 1}]

  def open!(path, write_mode \\ @write_mode) do
    File.open!(path, write_mode)
  end

  def close(file_descriptor) do
    File.close(file_descriptor)
  end

  def append!(file_descriptor, data) do
    {:ok, start_position} = :file.position(file_descriptor, :cur)

    data = :erlang.term_to_binary(data)
    :ok = IO.binwrite(file_descriptor, data)

    {:ok, end_position} = :file.position(file_descriptor, :cur)

    %{
      file_descriptor: file_descriptor,
      start_position: start_position,
      end_position: end_position,
      size: byte_size(data)
    }
  end

  def read!(path) do
    {:ok, data} = :file.read_file(path)
    data |> :erlang.binary_to_term()
  end
end

This example is a very simplified form of what this EventLog library does:

2 Likes

EventStoreDB is a custom-built event store which provides very high performance. There’s an Elixir library named extreme which can be used as a client. This is different from the Postgres-based Elixir EventStore library that I wrote (which it was inspired by). Both event stores are also compatible with Commanded and you can choose to use either store. Commanded provides a CQRS framework for event sourcing.

There is now a cloud hosted version named Event Store Cloud which allows you to deploy managed EventStoreDB clusters in the cloud through AWS, GCP, and Azure. I’d recommend looking at this if you want the best possible performance. The only caveat is that EventStoreDB is migrating to using gRPC as its client interface which is not well supported in Elixir. The existing extreme library uses the older TCP interface.

2 Likes

It seems to be this one:

Does it allow other storage outside of PostgreSQL?

<<1,2,3>> is a 3 byte binary: 0x01, 0x02, 0x03
"123" is 3 byte binary: 0x31, 0x32, 0x33

3 Likes

<<“hello”>> == <<“h”,“e”,“l”,“l”,“o”>>

Returns true on iex, I got confused thinking with integers they will be converted into strings and subsequently concatenated. It’s clear that in this case the integers will simply be stored in successive bytes unlike it string counterparts which are concatenated.

b = << 1::size(2), 1::size(3) >> == <<9::size(5)>>

Does the above implies when you indicate a size either than 8, it forces the numbers to be stored together hence generating a single value?

EventStoreDB does not use Postgres, it uses its own internal storage format and writes directly to disk. The Elixir EventStore library (confusingly I named it similarly) does use Postgres.

1 Like

You can do the other way around as well

<<1::big-size(32)>> == <<0,0,0,1>>

So it is not limited to the “joining”. In general <<>> can encode 2 things in Erlang and Elixir:

  • bit strings which is set of bits that is stored as sequence of bytes, with last entry being possibly non 8-bit long
  • binaries which are bit-strings with last byte always full aka bit strings with bit length that is multiply of 8

<<"a", "b">> is special syntax that is inherited from Erlang, where binaries that contain only Unicode can be written ass <<"a">> which is the same as <<$a>> (Elixir syntax for that is <<?a>>).

1 Like

My confusion stems from the fact that I was thinking for e.g <<3>> represented the decimal number 3 rather than a unicode code point.