Hello everyone,
Preview of Meshx library providing service mesh support.
Selected Meshx features are demonstrated with asciinema recordings. Tmux terminals are executed on separate vms (h11, h12, h21, etc.), each vm is running Consul Agent locally.
E1. MeshxNode and MeshxConsul introduction.
First we start iex with epmd disabled and using MeshxNode protocol distribution (see top left corner terminal for iex command). MeshxConsul is configured as service mesh adapter for MeshxNode and will start “node service” in background.
We turn nodes into distributed running Node.start/1 and Node.set_cookie/2.
By default, MeshxConsul will use Consul Connect built-in proxy as a sidecar proxy; you can see proxy stdout in terminals.
We test node connectivity with Node.connect/1 and Node.list/0. Next we execute few functions exported by MeshxConsul:
- list/0 - list services started on the node,
- list_upstream/1 - list (node) service upstreams,
- info/1 - service config as registered with Consul,
- MeshxConsul.Proxy.info/1 - (node) service sidecar proxy info.
Finally we run spawn() and :erpc.call() to confirm erlang distribution compatibility with Meshx.
E2. MeshxNode: HA Erlang distribution.
We start one upstream client node :cli@h11 (right) and two node service providers :service@any on h12 and h21 hosts (left).
We Node.connect/1 :cli@h11 to :service@any “node service” and use :erpc.call/4 with :inet.gethostname/0 to know which node was connected by sidecar proxy round-robin: connected node service is running on h21 host.
After stopping :service@any on h21 with Node.stop/0 and running :erpc.call/4 again:
- Service down event is detected by service mesh for service on h21. Traffic is automatically redirected by sidecar proxy to remaining “service@any” healthy instance on h12 discoverable with Consul registry.
- Erlang executes handshake with new node, valid cookies are still required.
- We receive response from healthy “service@any” h12 instance.
E3. MeshxRpc introduction.
We start two nodes. First node :s1 (left) will provide service “service1”, second node :c1 (right) will consume upstream service “service1”.
First, we attach built-in pretty-printing (dev)Logger to :telemetry events using MeshxRpc.Common.Telemetry.attach/2 on our “service1” provider node.
Then we run MeshxConsul.start/4 to start new service “service1” with Envoy configured as a sidecar proxy. Starting a service involves inter alia: template parsing (Mustache), registration (Consul), running sidecar proxy (Envoy) and running service TTL health check worker (GenServer).
Next we start RPC server “used/implemented” in Example2.Server module and bind it to address generated earlier by MeshxConsul.start/4.
We execute similar bootstrapping process for service consumer. Instead of starting service we connect to upstream “service1” using MeshxConsul.connect/3.
Finally we run few Example2.Client functions to demonstrate basic MeshxRpc API.
User controlled MeshxRpc settings: msg block size (see telemetry “blocks”), msg block checksums, configurable serialization (telemetry “ser”/“dser”), configurable request auditing up to the socket/connection level (conn_ref, node_ref, svc_ref, req_ref, hsk_ref).
Library is using custom binary protocol and connection pooling.
E4. MeshxRpc: LB and HA.
We start upstream client node :c1@h11 (right) and two :s1@h12 and :s1@h13 nodes, which will provide “service1” service. Example2.Client RPC client started on :c1@h11 is configured with 20 pool workers.
On client node :c1@h11 we Task.async/1 spawn 20 instances of call_do_work/0 function:
def call_do_work(), do: MeshxRpc.Client.Pool.call(Example2.Client, :do_work, Node.self())
Spawned tasks are load balanced between available service providers by sidecar proxy (round-robin by default). After killing top left service node we repeat spawning call_do_work/0. Remaining part of animation probably doesn’t require explanation, one can see expected HA behavior.
Comments.
Hashicorp Consul features valuable in discussed here context: intentions, observability, gateways, ACL, encryption.
Meshx is written in Elixir. Everything except some minor syntactic sugar API parts specific to Elixir Kernel.use/2 should work with Erlang (not tested yet).
MeshxConsul is implementing Meshx.ServiceMesh adapter behavior requiring four callbacks. Future mesh adapter implementations using other service mesh applications are possible (Kuma, Istio, others?) and should be compatible with MeshxNode and MeshxRpc.
References.
Hashicorp Consul: home, docs, tutorials.
Envoy Proxy: home, docs.
HAProxy Connect Consul integration: github.
Meshx: TBA.
