Fl4m3Ph03n1x

Fl4m3Ph03n1x

Background

I have an OTP application that reads a CSV file and puts it into memory. After following some advice from this community I went with the following approach:

  • My app file has 2 modules: Populator and Reader. Populator opens the file and indexes its contents into a persistent_term table, while Reader is a client that queries said table.
  • Populator is a Genserver, that starts with state initializing. Once it loads the file into memory its state evolves to :ready or :error depending on the outcome.
  • Before making a request, the Reader checks the Populator’s state. If its :ready it queries the memory table, otherwise refuses the request.

Best practices

Now, the issue here is initialization. As a Genserver, my Populator has the file’s path hardcoded. This app is a library with, so following the good guidelines of libraries states that this is a terrible idea.

I could put the file path into a config file inside a config folder, but the directives for good libraries discourage this. They instead encourage me to have a function in the Populator called populate_memory (for lack of a better name) that upon receiving a file name, opens the file and fills the table.

Problem

But the problem with this approach is that if I do it that way, then the projects that use the library will have to deal with Engine failure on startup, and the code will endup looking like this:


defmodule FootbalInterface.Application do
  @moduledoc false

  use Application

  alias FootbalEngine

  require Logger

  def start(_type, _args) do

    children = []
    opts = [strategy: :one_for_one, name: FootbalInterface.Supervisor]

    case FootbalEngine.new(file_path) do
      {:ok, :indexation_successful} ->
        Logger.info("""
        Application launched successfully.
        Waiting for requests.
        """)
        Supervisor.start_link(children, opts)

      {:ok, :partial_indexation_successful, fails} ->
        Logger.warn("""
        Application launched with errors. Check the CSV file for:
        #{inspect fails}
        Waiting for requests.
        """)
        Supervisor.start_link(children, opts)

      {:error, :no_valid_data_to_save} ->
        Logger.error("""
        Failed to launch application. CSV file has no valid data.
        Shutting down.
        """)
        {:error, :invalid_csv}

      {:error, reason} ->
        Logger.error("""
        Failed to launch application. An unknown error occurred:
        #{inspect reason}
        Shutting down.
        """)
        {:error, reason}
    end

  end
end

Which has also been heavily discouraged by the community in previous posts.

Question

So, how do I follow best practices for libraries and still follow the community recommendations for my use case?

Showing Posts 1 to 6

LostKobrakai

LostKobrakai

My guidelines for config is as follows:

  1. If it’s functions, receive config via params
  2. If it’s processes, receive config via params given to the childspec
  3. If the above is not convenient for the default use case, additionally fall back to application env or other stateful configuration methods.

For your case I’m wondering why you need a process (genserver) at all. If all you do is either startup or not you don’t really need to check the genserver at runtime anymore. If things failed the app doesn’t start to begin with.

I’d go for:

with {:ok, result} <- Lib.populate(path) do
  case result do
    :indexation_successful -> Logger.info("Application launched successfully.")
    {:partial_indexation_successful, fails} -> Logger.info("Application launched with errors. Check the CSV file for: #{inspect fails}")
  end
  Logger.info("Waiting for requests.")
  Supervisor.start_link(children, opts)
end

You could add more logging to the failing part, but the not starting application does already cause logs to be generated. If you’d skip the logging for the success case it’s even more simple than that.

An alternative I can see, which doesn’t require wrapping Supervisor.start_link could be:

children = [
  {Lib.Populator, [path]},
  …
]
Supervisor.start_link(children, opts)

Where the Populator succeeds or fails to start depending on the result of loading the file. After a success the process could just hibernate. Here the population would be done at init/1 and block till finished and everything starting later can just use the data as needed.

If you don’t want to block on startup you can use the above as well, do the loading after init/1, but you essentially need to query the state of loading the file whenever you try to access it and also it’s not as straight forward in terms of stopping the application, as there might already be more things started.

Fl4m3Ph03n1x

Fl4m3Ph03n1x OP

For your case I’m wondering why you need a process (genserver) at all. If all you do is either startup or not you don’t really need to check the genserver at runtime anymore. If things failed the app doesn’t start to begin with.

This decision stems from this discussion:

Where I was advised to create a GenServer that monitors the file, and self-heals the applications once the file is corrected. With the file corrected, the GenServer then does nothing and in theory I could kill it, but for the time being I am leaving it be (for simplicity reasons).

An alternative I can see, which doesn’t require wrapping Supervisor.start_link could be:

I am already using this alternative, but then path needs to be hardcoded, which is against the guidelines for libraries. I honestly do not see a situation where I can abide for the best practices guidelines and the advice given by the community.

This is already being done and is expected. It’s a cheap operation and not a problem.

LostKobrakai

LostKobrakai

This kinda contradicts your snippet in the entry post, which does fail hard if the file could not be loaded, while self healing means you just start up not matter what and wait for it to heal. Maybe there’s some timeout, but you initially accept it to not work.

Maybe you could first define what should happen in case of errors. Should the application fail to start up, stop after a timeout, not fail at all and just deal with data not being available or should even the user of your application make the decision?

LostKobrakai

LostKobrakai

If that’s part of you library users’ supervision tree your library itself does not hardcode anything. And yeah, all my code examples would live in the users MyApp.Application, not your library. I missed specifying that.

Fl4m3Ph03n1x

Fl4m3Ph03n1x OP

Let me see if I get this correctly. The app that uses my library, would then have the following code:

defmodule FootbalInterface.Application do
  @moduledoc false

  use Application

  alias FootbalEngine

  require Logger

  def start(_type, _args) do

    children = [
        {FootbalEngine, "path_to_file"}
    ]
    opts = [strategy: :one_for_one, name: FootbalInterface.Supervisor]
  end
end

Correct?

This is actually what I have been trying to achieve, except I don’t quite know how to prepare my library to be used like that.

Questions

  1. Is there any documentation or tutorial you could point me to ?
  2. Would my FootbalEngine app still be an OTP application, or would it be a simple library?
  3. Would it still have an applixation.ex file with use Application?
  4. Would it simply returns the Supervisor in the Footbal.new function?

Would really appreciate some guidelines.

LostKobrakai

LostKobrakai

It would be a library without own supervision tree. It just has module(s), which can be started by other applications. There’s therefore no need for an application.ex in your library.

I’m not sure what supervisor you’d need in your library if FootbalEngine is a genserver.

— All posts loaded —

Where Next? Top

Trending in Questions Top

water
I’m aware of 2 such projects, are there others? git: GitHub - team-alembic/realworld: A fullstack Phoenix LiveView application with back...
New
katta
I having some trouble figuring out if I have set myself too strict of standards for my production server. Currently I can handle 75% of r...
New
peck
Hi everyone. My team and I have been working on a fairly modest app based around video streaming and chat, but we’ve landed a customer t...
New
bradley
I really like the adapter patterns that ecto, nebulex, waffle, etc. use and would love find something similar for a key management servic...
New
unaware8150
Hello folks! So at work, we are seeing some situations where we have to define some “fixed” strings that are used across the codebase in...
New
Alvinkariuki
How Can I Optimise Compile Time Dependencies I have been building an elixir application for about 2 years now. Many modules and files ha...
New

Other Trending Topics Top

budgie
A little off-topic, but I feel like people here have a good head on their shoulders. I used to be quite good at making software. Was luc...
New
KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
budgie
I love Elixir. It’s one of 2 programming languages I’ve ever fallen in love with. But I don’t use it anymore. Serverless was the promis...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews