Abstractions and reuse

Questions / Help
1

Hi all,

How can I make #1 and #2 abstract so any elixir application can
define their own implementation and “wire” them in a file such as application.ex?
Example:

  • RDBMS/NoSQL/File/Memory based implementation of get_all_restaurants()
  • Web, API, Console, WebService, Queue, … implementation of publish()

I’m only interested in solutions which allow concurrent testing
where each test can define their own stub safely.

Note that at no point I mentioned that Phoenix or Ecto are involved.
If they are, that would be by implementing these abstractions…

defmodule Domain.MakePoll do
  def run() do
    get_all_restaurants()
    |> Enum.shuffle()
    |> Enum.take(3)
    |> Poll.new()
    |> publish
  end
end
1 Like
2

This is so general of a question, I just want to give the correspondingly general answer “it depends” but I think perhaps we can find something more interesting to say.

At your level of abstraction the first thing you need to consider are the type signatures of these core variable domain steps (get_all_restaurants, publish). Yes, I know Elixir is dynamically typed, but types are still there, and you must consider them if you want to plug in different implementations of these domain steps. I can infer that get_all_restaurants/0 returns something Enumerable (perhaps a Stream of Restaurant structs). publish/1 takes what I assume is a Poll struct, but it’s unclear what it should return.

Once you have those type signatures, then I recommend you step back from the ivory tower and roll up your sleeves and implement a concrete example of each of your abstractions. (I’ve spent time in the ivory tower myself and it’s a comfortable place, but it’s prone to omit important information you’ll need). Once that’s working, then add in your abstraction layer. You’ll likely have Domain.MakePoll.get_all_restaurants/0 read a MFA (module, function, args) tuple from the application environment (or some other source) and Kernel.apply/3 it. Your concrete version would live someplace like RestaurantFetcher.Memory.get_all_restaurants/0.

With all that working, you pick your second concrete version to implement. That may give you feedback that your type signatures need tweaking (this is the stuff hard to determine from the ivory tower) which means revisiting your existing concrete implementations. Hopefully by the 3rd concrete implementation those type signatures will stabilize.

You may choose behaviours at this point and you may want additional callback functions for the concrete implementation modules (like the example RestaurantFetcher.Memory) to manage the lifecycle of any auxiliary processes they may need (which will likely be a no-op for some). Having those 3 concrete implementations will supply the information you need to make good choices.

4 Likes
3

Thanks for your answer and while I understand and agree with the approach (don’t generalize too early) I am also trying an inside out approach where I start with high level code like the snippet above.

The signature should be clear (they are to me at least :slight_smile: )

get_all_restaurants : void -> List Restaurants
publish : Poll -> void

As such, any concrete implementation is coupled to some concrete IO devices. Effectively they become plugins to the snippet above.

My post is about how we can plug them into the snippet above.

I am not without solutions but I may have become paralysed by too many options and none being actually convincing. Now to enumerate a few:

Option 1: make the snippet above a higher order function. The plugins are functions, they are passed as arguments to the snippet above. The caller of the snippet above becomes responsible to provide the plugins butI don’t believe the caller should have this responsibility. Imagining a phoenix controller for example, that’s not the job of a controller. However, this option does satisfy the test concurrency criteria explained above.

Option 2: implement the plugin as a module, ideally with an elixir behaviour and wire it with the snippet above via Config. This one is not safe wrt to test concurrency.

Option 3: doesn’t apply here, but when some data structure gets involved a protocol can also decouple the plugin from the snippet above. It’s rather elegant and safe wrt test concurency. (Done it before, can link to a post in this forum to show an example). It may become a bit tedious as each implemented usecase will require new protocols and new implementation (if the usecases are different, why would they share contracts/protocols?)

That’s all I have unfortunately and I’m hoping this is a false trichotomy.

4

Sometimes rephrasing the problem can help people help you, so the problem I’m confronted with is how to implement a CompositionRoot in elixir.

…and with that I bet someone will say “you are thinking in OOP, you should think in FP” :roll_eyes:

5

Not exactly, but patterns are not an end in themselves - they simply put a name to a common solution approach for commonly appearing problems. And OO being what it is has a lot of patterns for situations that aren’t easily approached in an OO environment.

So rather than focusing on implementing some pattern in Elixir I think it would be more helpful if you could shed some more light and detail on the actual, concrete problem that you are trying to solve and possibly even expand the scope of the description of the surrounding situation.

Given the information that you have already given @gregvaughn’s approach seems reasonable. My interpretation of his post leads me to a tactic that is often used with behaviours when testing ***.

Given your previous exposure with Mox I expect that you are throughly familiar with this approach to using behaviours. So I can only assume that you still have some misgivings with that particular solution implementation. Can you maybe elaborate on what your residual concerns are?

Edit: (***) as implemented in that post the alternate behaviour is “bound” during compilation. But as described in @gregvaughn’s post Application.get_env/3 can be used to retrieve information to complete the MFA tuples that are necessary to complete alternate function calls with apply/3 (similar to how it was used in your other topic).

6

An expanded example:

# file: lib/my_storage.ex
defmodule MyStorage do

  @config_key :my_storage_config

  def config_key,
    do: @config_key

  def init do
    %{:module => m, :init => f} = config = get_config()
    handle = apply(m, f, [])
    {config, handle}
  end

  def put({%{:module => m, :put => f} = config, handle}, value) do
    handle = apply(m, f, [handle, value])
    {config, handle}
  end

  def peek({%{:module => m, :peek => f}, handle}, default) do
    apply(m, f, [handle, default])
  end

  defp get_config() do
    Application.get_application(__MODULE__)
    |> Application.fetch_env!(@config_key)
    |> Map.new
  end

end

# file: lib/tuple_storage.ex
defmodule TupleStorage do

  def tuple_init do
    IO.puts(Atom.to_string(__MODULE__))
    nil
  end

  def tuple_put(_handle, nil),
    do: nil
  def tuple_put(_handle, value),
    do: {:value, value}

  def tuple_peek(nil, default),
    do: default
  def tuple_peek({:value, value}, _default),
    do: value

end

# file: lib/agent_storage.ex
defmodule AgentStorage do

  def agent_init do
    IO.puts(Atom.to_string(__MODULE__))
    {:ok, pid} = Agent.start(fn -> nil end)
    pid
  end

  def agent_put(pid, nil) do
    Agent.update(pid, fn _ -> nil end)
    pid
  end
  def agent_put(pid, value) do
    Agent.update(pid, fn _ -> {:value, value} end)
    pid
  end

  def agent_peek(pid, default) do
    case Agent.get(pid, fn contents -> contents end) do
      {:value, value} ->
        value
      _ ->
        default
    end
  end

end

# file: lib/demo.ex
defmodule Demo do

  def run() do
    handle = MyStorage.init()
    result = MyStorage.peek(handle, :undefined)

    IO.inspect(result)

    handle = MyStorage.put(handle, :something)
    result = MyStorage.peek(handle, :undefined)

    IO.inspect(result)
    :ok
  end

  def show do
    config =
      Application.get_application(__MODULE__)
      |> Application.get_env(MyStorage.config_key, :empty)

    IO.puts("Config: #{inspect config}")
  end

  def load(config) do
    Application.get_application(__MODULE__)
    |> Application.put_env(MyStorage.config_key, config)
  end

  def agent_config,
    do: [module: AgentStorage, init: :agent_init, put: :agent_put, peek: :agent_peek]

  def tuple_config,
    do: [module: TupleStorage, init: :tuple_init, put: :tuple_put, peek: :tuple_peek]

end

$ iex -S mix
Erlang/OTP 21 [erts-10.0.7] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Compiling 4 files (.ex)
Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Demo.show()
Config: :empty
:ok
iex(2)> Demo.load(Demo.tuple_config())
:ok
iex(3)> Demo.show()
Config: [module: TupleStorage, init: :tuple_init, put: :tuple_put, peek: :tuple_peek]
:ok
iex(4)> Demo.run()
Elixir.TupleStorage
:undefined
:something
:ok
iex(5)> Demo.load(Demo.agent_config())
:ok
iex(6)> Demo.show()
Config: [module: AgentStorage, init: :agent_init, put: :agent_put, peek: :agent_peek]
:ok
iex(7)> Demo.run()
Elixir.AgentStorage
:undefined
:something
:ok
iex(8)>
2 Likes
7

:+1: I can try to address things better now that I understand the core of your conundrum.

How very FP of you. but … (see below)

This was effectively my suggestion above of where to start just because it’s easy and straightforward. But you are absolutely right about the test concurrency angle. It wasn’t clear to me whether you needed this level of config at an app level or multiple combinations of plugins in a single app.

I don’t think protocols are ideal here either.

I’ve been trying to internalize wisdom I’ve received, the most key pieces being Joe Armstrong’s “Concurrency Oriented Programming” idea and Franceso Caesarini saying “processes should be used for each unit of concurrency in your system”. So I’ll offer a new option.

Not me :grin: . A great thing about concurrency-orientented thinking is that there is an overlap with the good parts of OOP.

Option 4: make the snippet above a process (either GenServer or Task probably) if it’s proper to say “poll publication” is a unit of concurrency in your system. The plugins (as MFA) are state in the process. This solves your testing concurrency requirement.

There’s also an Option 4a that I’ve been using that I am quite pleased with. You could refactor to it if/when it makes sense or start here. Use a struct to contain the context of the processing by taking inspiration in how Ecto.Query, Ecto.Changeset and Plug.Conn work. Your concrete plugin steps are stored in the struct when it is initialized then the struct is threaded through your logic (vasty changing your type signatures :slightly_frowning_face:). This allows you to decouple the processing logic from the concurrent process model. I use the separation to throttle outgoing 3rd party API calls. This approach is covered well in this talk.

8

Writing these abstractions can be done by just writing modules with the same functions.

However, to be more certain that your implementations follow the same behaviour, it probably makes a lot of sense to actually define a Behaviour for them: In this way the Elixir compiler is able to help you somewhat with building them. And besides this, it is a way to make it very visible to other developers (including your future self) that multiple implementations follow the same specification.

It is very common to see these kinds of implementations in applications. For instance, many applications and/or libraries create a super-simple implementation (that is either slower or does not do all of the required work) which can readily be used in unit tests (that test the other parts of the system), or be used to compare in tests the actual implementation with this simple one (This is one way in which fuzzing e.g. QuickCheck can work well).

1 Like
9

Indeed, I forgot the token based approach that you describe in 4a. I will give it more thought.

Independently of the nature of the plugins (a function, a module, a struct+protocol, a process, or a token, or a mix!), they must be declared somewhere in the system. I believe that place to be the “main” file. In Elixir, this would be the Application module. Ultimately the high level code like my first snippet must somehow use the declared plugins. The “wiring” is where I’m particularly struggling.

For example, let’s consider a web app independently of the nature of our plugin(s):

  • We decide to rely on the file system for reading and writing data
  • Application declares a filesystem based storage plugin
  • a http request enters the system and is routed to a controller by phoenix
  • the controller does some web related work
  • the controller delegates to some high level code like my snippet

How does the snippet get the storage plugin? Let’s consider it’s provided as an argument. That means the controller (the caller) has to get the storage plugin. How/where does the controller get the storage plugin?
And if the plugin is not provided as an argument, then it must come to the snippet in some other way. A way that would quite possibly be unsafe wrt tests concurrency.

10

I’m confused again. I initially assumed you would use one instance of a plugin in a given app, for which the application environment config is a natural choice. Then you explained about needing more dynamic choice so we went down the path of more complex design choices. The more dynamic situation would expect the controller to provide arguments to the snippet.

I see two situations:

  1. If different controllers need different combinations of plugins, then they just need to contain that information and pass arguments to the snippet. The different choices of plugins could be stored in different keys in application config if you don’t want a hard compile time dependency.

  2. If all controllers in an app need to use the same set of plugins, then either they retrieve them from the same place in the application environment and pass as arguments to the snippet, or else the snippet reads them from application environment.

Perhaps what you want is as simple as having your snippet take an optional parameter(s)? If the parameter(s) is not provided then the snippet looks it up from application environment. Then you don’t have duplicate lookup code in your controllers but your tests can explicitly (and concurrently) supply them to the snippet via parameters.

1 Like
11

I’m not opposed to a non dynamic solution as long as I can safely stub the plugins in my concurrent tests. Application Config not only has limitations (you can’t do config :app, storage: fn thing -> ... end) but it also limits by design the declaration of a plugin to a unique implementation at any point in time (Application.get_env :app, :storage_plugin can only have one value at any point in time or concurrent tests would step over each other).

Initial thought, no. Second thought, oh wait, maybe, ah, oh…

Saving this one for later, simple and effective. However, I’m not a big fan of using Application config anymore, I’d rather limit its use for static values required at compile time by the compiler which is rare. As I explained above, once you rely on config for injection, it can’t be swapped dynamically in concurrent tests.

I suppose that “application environment” means application config (which you used in point 1)? Then point 1 is for example:

config :app, :controller1, %{storage_plugin: foo1, notification_plugin: bar1}
config :app, :controller2, %{storage_plugin: foo2, notification_plugin: bar2}

and point 2 is

config :app, storage_plugin: foo
config :app, notification_plugin: bar

along with an adequate Application.get_env in the controller or snippet?

Indeed. I can’t think of another solution with the current state of Elixir.

12

By the way, thanks all for your help, especially @gregvaughn

13

@svarlet You are wise to be cautious about config, because those are evaluated at compile time, but don’t avoid it so much to make more work for yourself. It’s also important to remember that even though config is executed at compile time, it is put into the “application environment” ETS table at runtime, and that can be modified at runtime with Application.put_env/3 So, it may be splitting hairs, but application environment is more dynamic than config. It is still effectively a global variable though.

Yes, except that I would make foo and bar a tuple of {Module, function} atoms.

14

Also, something that many libraries do, is to have the ‘application default’ configured in the config, but allow you to override this default by using an extra explicit argument to the invocation function.

As an example, consider my okasaki queue library: You can instantiate a queue using Okasaki.Queue.new([1,2,3]), which will use the library-default queue implementation. If your configuration specifies a different one, however, then that one is used by default instead.
And if you want to use a different one (which is e.g. used inside the tests of the library, and is also exactly what is done when creating benchmarks for these kinds of libraries), you can use Okasaki.Queue.new([1,2,3], implementation: Okasaki.Implementations.ConstantQueue).

1 Like