ETS update_counter error

Still new to elixir and exploring so here is something I could use some help with

I have an ets with following definition

  :ets.new(table, [
          data_type,
          :public,
          :named_table,
          read_concurrency: true,
          write_concurrency: true
        ])

All are initializsed as follows upon application startup

    :ets.insert(table, {:uptime, DateTime.utc_now() |> DateTime.to_string()})
    :ets.insert(table, {:total_msg_count, 0})
    :ets.insert(table, {:replied_msg_count, 0})
    :ets.insert(table, {:forwarded_msg_count, 0})
    :ets.insert(table, {:total_attachments_downloaded, 0})
    :ets.insert(table, {:total_messages_slacked, 0})

Here is a my code that I use to manage ets in my project.

require Logger

defmodule Ultronex.Realtime.TermStorage do
  @moduledoc """
  Documentation for Ultronex.RealtimeTermStorage
  """
  def initialize do
    ets_initialize()
    initialize_stats()
  end

  def initialize_stats do
    output = ets_initialize(:stats, :set)

    case output do
      {:error, _} ->
        Ultronex.BotX.ets_initialize()

      _ ->
        Logger.debug(":ets : stats loaded from file")
    end
  end

  def ets_initialize(table \\ :track, data_type \\ :bag) do
    Logger.info("Creating :ets : #{table} , type : #{data_type}")

    output = :ets.file2tab('tab/#{table}.tab')

    case output do
      {:error, _} ->
        Logger.info(":ets : #{table} created from fresh")

        :ets.new(table, [
          data_type,
          :public,
          :named_table,
          read_concurrency: true,
          write_concurrency: true
        ])

      {:ok, _} ->
        Logger.info(":ets : #{table} created from file")

      _ ->
        Logger.debug(":ets : #{table} - WTF !!!!!!")
    end

    output
  end

  def ets_incr(table \\ :stats, key \\ :total_msg_count) do
    :ets.update_counter(table, key, {2, 1})
  end

  def ets_lookup(table \\ :stats, key \\ :total_msg_count) do
    :ets.lookup(table, key)[key]
  end

  def ets_tab2file(table) do
    Logger.info("Saving.... :ets : #{table} to file")
    :ets.tab2file(table, 'tab/#{table}.tab')
  end
end

The error happens for the following key :total_messages_slacked only on update_counter and that to randomly from the looks as its not happening on all updates
All are updated in following manner

TermStorage.ets_incr(:stats, :replied_msg_count)
TermStorage.ets_incr(:stats, :forwarded_msg_count)
TermStorage.ets_incr(:stats, :total_messages_slacked)

The additional thing to know is that :total_messages_slacked is updated by a different process than the one that owns the ets table, but as its public it should work and does like 99% of the time.

coming to the error that is happening and :total_messages_slacked is only updated in one place

(Protocol.UndefinedError) protocol String.Chars not implemented for {%ArgumentError{message: "argument error"}, [{:ets, :update_counter, [:stats, :total_messages_slacked, {2, 1}], []}, {Ultronex.Realtime.TermStorage, :ets_incr, 2, [file: 'lib/ultronex/realtime/term_storage.ex', line: 52]}, {Ultronex.BotX, :relay_msg_to_slack, 4, [file: 'lib/ultronex/bot_x.ex', line: 77]}]} of type Tuple. This protocol is implemented for the following type(s): List, Date, Version, URI, Integer, BitString, Time, Version.Requirement, Float, DateTime, NaiveDateTime, Atom

Wierd thing is not sure what its complaing about at times as its not possible for the value to be of different type suddenly for a process.

The error you posted is hiding somewhat the underlying badarg from the :ets.update_counter/3 call. In the docs it says this can happen when any of:

  • The table type is not set or ordered_set.
  • No object with the correct key exists and no default object was supplied.
  • The object has the wrong arity.
  • The default object arity is smaller than .
  • Any field from the default object that is updated is not an integer.
  • The element to update is not an integer.
  • The element to update is also the key.
  • Any of Pos, Incr, Threshold, or SetValue is not an integer.

The error is happening, according to your error message, on a call to:

:ets.update_counter :stats, :total_messages_slacked, {2, 1}

Of course that doesn’t identify why the error is happening. Perhaps one step would be to rescue the badarg and dump the ets table since its quite small. Perhaps there will be a clue in the data?

  def ets_incr(table \\ :stats, key \\ :total_msg_count) do
    :ets.update_counter(table, key, {2, 1})
  rescue ArgumentError ->
    IO.puts "Exception: ets_incr called with table: #{inspect table} and key: #{inspect key}"
    IO.inspect tab2list(:stats)
  end

At least that might give you enough information to work out what the issue is.

2 Likes

thanks @kip i will try this out and share my findings.

seems the issue effects the ETS at that given moment as I can’t even get it to do

IO.inspect(:ets.tab2list(:stats))

as even that throws following exception in rescue

(ErlangError) Erlang error: {:EXIT, {%Protocol.UndefinedError{description: "", protocol: String.Chars, value: {%ArgumentError{message: "argument error"}, [{:ets, :match_object, [:stats, :_], []}, {:ets, :tab2list, 1, [file: 'ets.erl', line: 763]}, {Ultronex.Realtime.TermStorage, :ets_incr, 2, [file: 'lib/ultronex/realtime/term_storage.ex', line: 58]}, {Ultronex.BotX, :relay_msg_to_slack, 4, [file: 'lib/ultronex/bot_x.ex', line: 77]}]}}, [{String.Chars, :impl_for!, 1, [file: 'lib/string/chars.ex', line: 3]}, {String.Chars, :to_string, 1, [file: 'lib/string/chars.ex', line: 22]}, {Logger.Formatter, :"-output/5-fun-0-", 1, [file: 'lib/logger/formatter.ex', line: 166]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1336]}, {Logger.Formatter, :"-format/5-fun-0-", 6, [file: 'lib/logger/formatter.ex', line: 152]}, {Enum, :"-reduce/3-lists^foldl/2-0-", 3, [file: 'lib/enum.ex', line: 1948]}, {Logger.Formatter, :format, 5, [file: 'lib/logger/formatter.ex', line: 151]}, {LoggerFileBackend, :log_event, 5, [file: 'lib/logger_file_backend.ex', line: 69]}]}}

I switched clouds provider and still see it but the number has reduced for now I will term this as an issue where memory is not readily available i cloud and this is caused. Would be nice to know if ppl have faced such issue with cloud providers while using ETS. thanks for your help @kip, learned a couple of new things.

It seems I might have finally solved the issue by making 2 changes

  1. Move ETS to its own GenServer and not manages it from another.
  2. Don’t update counter async and write ETS to file for snapshot.

Really appreciate the community and @kip help