Some questions about working with multiple connected processes, linking and app structure

[mesuutt]

Imagine such a scenario:

There are 2 companies(dealer and supplier), each one using their own websites.
I want to create a service between them.

My service will periodically fetch new product request from the dealer’s API and send the product request to the supplier’s API. Also, the service will periodically check sent product requests from supplier’s API and if there is a change about the sent products on the supplier’s website the service will send related changes to dealer’s API.

I don’t know exactly what is the best way of handling this type of scenario.

What do you think about building a structure like this:

Start 2 GenServers for each dealer<=>supplier binary when application started.

I think creating two separate genserver will provide me separation of dealer/supplier related code from each other.

DealerClient:

- Fetch new products from dealer API(periodically)
- Send product changes sent by supplier genserver to dealer API.

SupplierClient:

- Fetch product changes from supplier API(periodically).
- Send the changes to dealer genserver.

So data flow will be like this:

DealerAPI <-> DealerClient <-> SupplierClient <-> SupplierAPI

Also there are multiple dealers and multiple suppliers(OneToOne relation), so each dealer process should know
her supplier’s process to send or fetch data from.

Questions:

• This structure is good? If not what would you do if you were me? What kind of structure would you build?
• How to handle dynamic process linking? How does dealer process know her supplier’s process?
• Should I create a supervisor for each dealer<=>supplier binary?

Sorry about my poor English.

[kokolegorille]

Hello, welcome to the forum.

That is a nice project, let me have my 2 cents opinion…

GenServer are not object, the closest You might find in Elixir is a struct. A struct is like an object without method, what You would call methods, are Module functions…

You can learn about good usage of GenServer here https://www.theerlangelist.com/article/spawn_or_not.

What about a GenServer called LinkService, where You would pass dealer and supplier as params Structs?

You have probably many ways to do it, but I don’t think You should model your business directly with GenServer

[mesuutt]

@kokolegorille

What about a GenServer called LinkService, where You would pass dealer and supplier as params Structs?

Should come 10 product per second sometimes. Does generating only one process for all work block concurrency? (Maybe I can start a task in the process for every fetched product to send to supplier for don’t block process. Does this good?)

@kokolegorille

You have probably many ways to do it, but I don’t think You should model your business directly with GenServer

Thanks for the advice. I will separate business logic from GenServer.

[kokolegorille]

I meant one GenServer per dealer/supplier…

I would use Dynamic supervisor for this, with a Registry. This would allow to have multiple processes, and easy access through keys.

[tty]

Short answer: you structure is good and sufficient.

My suggestion would be one application per Dealer. Under the application have the standard supervisor. Under this standard supervisor one GenServer which does the dealer polling. Under the same supervisor have another GenServer that does the supplier polling.

Assumption: Supplier to Dealer is a 1-1 relation.
Assumption: New product requests are periodic and low volume.

There is absolutely nothing wrong with what you had suggested.

The reason for my suggestion is isolation. It makes it easier to migrate Dealers to a different node or upgrade a Dealer since it is fully encapsulated within its own application.

[lud]

Welcome there

I think you misuses GenServer for objects (as in OO). Kind of what @kokolegorille said.

I think it would be easier to desing and to test with Task, or other short lived processes.

My service will periodically fetch new product request from the dealer’s API and send the product request to the supplier’s API

When you say “periodically” and then Should come 10 product per second sometimes … How is a fetch triggered ? Is it from external input or do you have a process that triggers the fetch every X seconds ?

I think I would just write the code that does 1) fetch a new product request and 2) send it to the supplier. When it works, then
I would just wrap that in a Task.
If your fetch returns multiple products, I would spawn a Task for each one or use Task.async_stream to limit concurrency and to be gentle with the external endpoint.

What about a GenServer called LinkService, where You would pass dealer and supplier as params Structs?

This GenServer could be the one that does the fetch every N seconds or listens for external inputs ; and spawn tasks.

[kokolegorille]

if dealer and suplier have distinct tasks, it is absolutely valid to have one process for each. I just wanted to warn about designing OO around GenServer

[lud]

Well you need to fetch something before you can send it to the other API so those things are not concurrent.

There is the reverse feature which I did not address, the Supplier => dealer fetch and send, which is independent of the first feature (as far as we know) so there another process is good. Is that what you meant ?

[kokolegorille]

I wanted to respond to @tty post

That is also why my first proposal was to have only one. But the real answer is…

it depends of requirement

[mesuutt]

@tty

My suggestion would be one application per Dealer.

Can I start/stop applications dynamically with different configs(API tokens or dealer/supplier ids)?

What is the your preferred way communicate dealer process with supplier API?(over supplier process or direct request to supplier API)

I didn’t understand what is polling.

I created a dynamic supervisor which has 1 dealer genserver and 1 supplier genserver.
I am binding processes in supervisor

defmodule MySupervisor do
  use DynamicSupervisor

  def start_link(_arg) do
    DynamicSupervisor.start_link(__MODULE__, :ok, name: __MODULE__)
  end

  def init(:ok) do
    DynamicSupervisor.init(strategy: :one_for_one)
  end

  def start_workers(%{:supplier_token => supplier_token, :dealer_token => dealer_token} = params) do
    {:ok, dealer_pid}  = supervise_dealer_server(dealer_token)
    {:ok, pgw_pid}  = supervise_supplier_server(supplier_token, dealer_pid)
    {:ok, pgw_pid}
  end

  defp supervise_supplier_server(supplier_token, dealer_pid) do
    spec = %{id: MyApp.SupplierWorker, start: {MyApp.SupplierWorker, :start_link, [%{:token => supplier_token, :dealer_pid => dealer_pid}]}}
    DynamicSupervisor.start_child(__MODULE__, spec)
  end

  defp supervise_dealer_server(dealer_token) do
    spec = %{id: MyApp.DealerWorker, start: {MyApp.DealerWorker, :start_link, [%{:token => dealer_token}]}}
    DynamicSupervisor.start_child(__MODULE__, spec)
  end
end

After start dealer/supplier processes I am binding processes shown as below:
In DealerWorker

  def handle_call(:bind_supplier, from, state) do
    state = Map.put(state, :supplier_pid, from)
    {:reply, state, state}
  end

In SupplierWorker.init

    GenServer.call(dealer_pid, :bind_supplier)

@lud

How is a fetch triggered ? Is it from external input or do you have a process that triggers the fetch every X seconds ?

I am triggering fetch in processes with Process.send_after(self(), :check, 1_000 * 5). Not from any external input. I don’t know this is good because what it do if process not finish work after 5 seconds(for now).

@lud

I would just wrap that in a Task.

This is good idea.

[lud]

@mesuutt

I am triggering fetch in processes with Process.send_after(self(), :check, 1_000 * 5) . Not from any external input. I don’t know this is good because what it do if process not finish work after 5 seconds(for now) .

If this is a GenServer, it will receive this message after 5 seconds and if it is busy at the moment, the current work will be finished before your handle_info function is called. So no problem with that. And you can then call send_after again in handle_info.