Logging too many log request

You don’t need to learn binary to save to a log file, you just use :erlang.term_to_binary and then pass it into IO.binwrite(file_descriptor, binary_data) to save into the file and when reading from the file you jsut use :erlang.binary_to_term to get your data decoded:

defmodule FileIO do

  @moduledoc """
  iex(3)> fd = FileIO.open! "test.bin"
  #PID<0.200.0>
  iex(4)> FileIO.append! fd, %{key: "value"}
  %{end_position: 22, file_descriptor: #PID<0.200.0>, size: 22, start_position: 0}
  iex(5)> FileIO.read! "test.bin"
  %{key: "value"}
  """

  @write_mode [:append, :binary]
  # @write_mode [:append, :binary, :delayed_write]
  # @write_mode [:append, :binary, {:delayed_write, 1, 1}]

  def open!(path, write_mode \\ @write_mode) do
    File.open!(path, write_mode)
  end

  def close(file_descriptor) do
    File.close(file_descriptor)
  end

  def append!(file_descriptor, data) do
    {:ok, start_position} = :file.position(file_descriptor, :cur)

    data = :erlang.term_to_binary(data)
    :ok = IO.binwrite(file_descriptor, data)

    {:ok, end_position} = :file.position(file_descriptor, :cur)

    %{
      file_descriptor: file_descriptor,
      start_position: start_position,
      end_position: end_position,
      size: byte_size(data)
    }
  end

  def read!(path) do
    {:ok, data} = :file.read_file(path)
    data |> :erlang.binary_to_term()
  end
end

This example is a very simplified form of what this EventLog library does:

2 Likes

EventStoreDB is a custom-built event store which provides very high performance. There’s an Elixir library named extreme which can be used as a client. This is different from the Postgres-based Elixir EventStore library that I wrote (which it was inspired by). Both event stores are also compatible with Commanded and you can choose to use either store. Commanded provides a CQRS framework for event sourcing.

There is now a cloud hosted version named Event Store Cloud which allows you to deploy managed EventStoreDB clusters in the cloud through AWS, GCP, and Azure. I’d recommend looking at this if you want the best possible performance. The only caveat is that EventStoreDB is migrating to using gRPC as its client interface which is not well supported in Elixir. The existing extreme library uses the older TCP interface.

2 Likes

It seems to be this one:

Does it allow other storage outside of PostgreSQL?

<<1,2,3>> is a 3 byte binary: 0x01, 0x02, 0x03
"123" is 3 byte binary: 0x31, 0x32, 0x33

3 Likes

<<“hello”>> == <<“h”,“e”,“l”,“l”,“o”>>

Returns true on iex, I got confused thinking with integers they will be converted into strings and subsequently concatenated. It’s clear that in this case the integers will simply be stored in successive bytes unlike it string counterparts which are concatenated.

b = << 1::size(2), 1::size(3) >> == <<9::size(5)>>

Does the above implies when you indicate a size either than 8, it forces the numbers to be stored together hence generating a single value?

EventStoreDB does not use Postgres, it uses its own internal storage format and writes directly to disk. The Elixir EventStore library (confusingly I named it similarly) does use Postgres.

1 Like

You can do the other way around as well

<<1::big-size(32)>> == <<0,0,0,1>>

So it is not limited to the “joining”. In general <<>> can encode 2 things in Erlang and Elixir:

  • bit strings which is set of bits that is stored as sequence of bytes, with last entry being possibly non 8-bit long
  • binaries which are bit-strings with last byte always full aka bit strings with bit length that is multiply of 8

<<"a", "b">> is special syntax that is inherited from Erlang, where binaries that contain only Unicode can be written ass <<"a">> which is the same as <<$a>> (Elixir syntax for that is <<?a>>).

1 Like

My confusion stems from the fact that I was thinking for e.g <<3>> represented the decimal number 3 rather than a unicode code point.

Binary by itself has nothing to do with unicode.
<<3>> is the binary number 3 in a byte, which is also decimal 3.
If you want the unicode code points you have to do something like <<"☺"/utf8>> in erlang or “:slight_smile:” in elixir, which is <<226,152,186>> deep down.

1 Like

I need a bit of education/clarification on; e.g <<0>> == "\0" and <<0>> != 0 or it string equivalent "0", same goes for <<7>> == "\a" and <<7>> != 7.

Is there something under the hood that I’m missing?

A binary is a sequence of arbitrary bytes. An elixir string is a binary representing the utf8 encoded string.
All strings are binaries, not all binaries are strings.
If you want to use a string literal, do not write it in binary form like <<…>> because it will confuse the hell out of readers. Use a double quoted string.
If you want to use a binary literal that is not a legal utf8 string, write it in binary form with numbers 0~255 to avoid confusions like:
<<354>> == <<98>> == "b"

4 Likes

https://stackoverflow.com/questions/36094791/shell-represents-binary

This link was very helpful.

1 Like