Async TCP subscription through a library, without GenServer

pub/sub

What are you talking about?

Now, we think a GenServer would cause a whole lot of unnecessary complexity

Why is that? Consider http clients for example. Using any library with processes is much easier than implementing mint message handling.

So, I’d suggest to create a process for this, it’ll be easier to use and this can always be optimized later in case you find an extra process a bottleneck you can’t afford (but I think you won’t).

Anyway, if you don’t want to you can take this genserver approach, you can just have something like this

def connect(...) do
  ...
  {:ok, state}
end

def handle_message(state, message) do
  # Here we check if the message is for our library
  # and return `{:ok, result, new_state}` if it is,
  # or {:error, :bad_message} if it's not
end

I’d suggest to take a look at Mint to see how they handle it there.

But I’d still consider using processes

If you don’t want to use GenServer you can just spawn on the spot and remember the client’s pid for sending back the data.

Well, I might be wrong. Hence the forum message.

The real use case is a tad more involved. This service, while encapsulated by the lib and now easier to use, still returns pretty raw data, message contents should be kept as is. It could be seen as the data layer in a three tier app.

Another lib would be responsible for business logic and further data refinement. The programmer using these libs shouldn’t have to care or even know about the data lib, but keep focus on the business logic one.

If the data lib has a GenServer, I don’t think there’s any way around the fact that the business logic lib should also have one. This is to prevent details about the data lib to filter through. In that case, the business logic GenServer should be responsible for the data lib GenServer and thus may end up being a Supervisor, I guess.

Nevertheless, in this situation, there are 3 processes at a minimum: the inner lib, the business logic lib, and the program itself.

This situation could be the way to go… but it still has several implications. One that bothers me is the fact that data would have to be copied twice before it gets in the program’s process. Might slow things up for larger chunks of data…

I am just wondering about best practices to architect such a thing and am very interested in what the community thinks about it.

Not a bad idea, and that would fix the copying problem mentioned in the post above.

We are unsure about the ease of use of the said lib, though…

Any experienced elixir system architect’s opinion would be welcome here.

I don’t understand a word you’re saying. Let’s start with something simple
What are you trying to acheive? And what API do you want to provide?

The data comes from an Electrum RPC endpoint, which is what most Bitcoin wallets use to transact.

For instance, the blockchain.scripthash.subscribe is mostly what I am talking about. So, the data lib I was talking about would connect to this endpoint and make sure it subscribes to a specific scripthash.

But what’s a scripthash???

It’s jargon that Bitcoin wallet devs might not want to deal with. In other words, it’s another representation of an address. This call, in short, returns transactions relative to a specific Bitcoin address as they happen.

The goal here is to not only enable devs to connect to this thing, but make it easy and relatable. Ideally, they’d just initiate a connection, passing an IP and a port and then subscribe to a specific address. This is, of course, in the Bitcoin layer lib’s language. Under the hood, it would handle the electrum lib and isolate the dev from all the scripthash gibberish we initially talked about. Not to mention there’s some data conversion involved in the process.

Why not combine both in a single lib, you might ask… because the raw Electrum lib

• Could be used for other purposes someday
• Would be easier to maintain if kept 1:1 with the Electrum RPC specification

From what you wrote I can see that you want to different libraries:

1. Lib just for Electrum RPC protocol. You want it process-less
2. User-friendly Electrum interface with more high-level operations

So I can see 3 layers in this program

1. TCP connection pool management. To connect to peers and communicate with them.
2. Electrum JSONRPC protocol
3. High-level API for the user (without scripthash, etc)

The connection management

Alright, I’ve read the python library and I can see that these RPC calls operate over some state (SessionManager class holds global state of all sessions, and connection to a PeerManager which maintains TCP connection to max 8 peers). And you’ll need to handle this state somehow. In erlang and elixir world we use processes for that.

And there are two options for the process. You can make it a global singleton, or a local process. If you want to maintain exactly the same API as in ElectrumX, you’ll need to create a named singleton process. But I’d suggest making library more flexible and isolating process names from it. So, I’d go with SessionManager as a Supervisor module which starts DynamicSupervisor, Registry and starts ElectrumConnection processes under the DynamicSupervisor. This SessionManager can take a name during the start. Each ElectrumConnection holds connection to one peer

And the module would provide these functions

• SessionManager.add_peer(session_manager_pid, peer_opts) to add a peer. This will start the new connection process under the dynamicsup, name it in the registry.
• SessionManager.connect_peer(), SessionManager.disconnect_peer, etc. for peer management
• SessionManager.pick_connection() – randomly picks one connection (you can chose different strategy)
• SessionManager.execute(connection, operation, reply_to) – this one just passes an Electrum operation to the connection process which should send the result to reply_to pid

This is basically how the process pool operates. I can suggest you to take a look at libraries like poolboy

Electrum JSONRPC

This part of the system is a thing that you wanted to be a process-less library for just parsing and handling responses. The problem with your approach was that you wanted it to be somehow coupled with tcp connection, which is a bad way to go.

I’d suggest spliting this into two things:

1. Converting elixir structure for operation into the binary with jsonrpc request/response in it. This is a pretty straightforward thing to write. You can use any data validation library you prefer, but I don’t see any problem in hand-rolling this.

2. Sending and receiving data over TCP as a JSONRPC spec defines. This was already done in libraries like this: GitHub - fanduel-oss/jsonrpc2-elixir: JSON-RPC 2.0 for Elixir · GitHub

Highlevel user interface

Just create something like Electrum.subscribe which internally calls already started global SessionManager and awaits the response

I appreciate the answer, thanks!

Just one detail… there are no peer in the problem. We just subscribe to a server and want to manage realtime data that comes out of it.

The solution we’re about to do is pretty much what you said in the beginning. We will probably keep the GenServer in the electrum lib and use a Supervisor in the business logic lib. That would allow the business logic lib to connect to other GenServers in the same way, including a potential wrapper around Bitcoin Core.