So, I haven’t really worked with MQTT before and now I want to build a proof of concept project using it.
Some context:
We have many devices and today we are not using MQTT, but we might do in the future. Our API makes HTTP requests to devices. We don’t want the API itself to handle MQTT but keep doing HTTP requests.
So basically:
API --HTTP POST–> Elixir App --MQTT–> Broker --MQTT–> Device
and for state changes etc:
Device --MQTT–> Broker --MQTT–> Elixir App → API/Google PubSub or wherever it should be piped.
I have a basic setup now with a simple ESP that connects to a broker and an elixir app which spawns a Genserver (with emqtt) which connects to the broker and subscribes to some topics. This works fine for my single test device.
My feeling though is that it might be a bit limiting to have one Genserver handling all MQTT messages, right?
How does one normally handle this when you have thousands of devices? Spawn a Genserver per device which only handles that device’s messages?
Keep it at one Genserver but as soon as a message arrives, spawn a new process to handle it?
I guess it also depends on the frequency of messages per device as well, and each device won’t send a lot of messages. There might be times during the day where many devices send messages at about the same time, but each device won’t generally send messages at a high frequency.
Anyway, I know that it’s probably very application specific, but perhaps someone with more MQTT experience can give some more general tips and guidelines?
Since you will most probably will not write the MQTT server yourself, you should take a look at existing libraries and if they fit your needs (for example I am not sure all of the existing libraries support MQTT3).
The part responsible for concurrency should be already implemented in the library in most cases, so I would say to first investigate existing libraries.
With MQTT server you mean the Broker? I have already picked EMQX and I’m using their Erlang mqtt client emqtt (it support MQTT5 as well, which I need since I want to test out the request/response pattern built into v5).
My test project so far is inspired by their guide MQTT for Elixir | EMQ and I have it connected already. I can publish/subscribe in my ESP8266 and I can publish/subscribe in my Elixir app
Now, in their example they spawn a genserver, in which they connect to the broker and all messages arrive in that genserver as calls to handle_info.
So my question is mainly, is that a valid approach for larger fleets of devices or what should I keep in mind to make it scalable?
OK, it has been some time since I worked with embedded devices , the last time I worked on a embedded project v3 spec was recently out.
OK, this makes more sense.
Highly depends on what kind of QOS you are aiming to use. If you aim to use QOS 0, then you can most probably get away with a single genserver (as long as you don’t do the processing in that process too).
Since processes are cheap to spawn, I would recommend for the first implementation to go wild and handle as much as you can in separate processes. You can wire some telemetry at the same time and measure the actual throughput to determine if the solution is an overkill and optimize accordingly.
, the last time I worked on a embedded project v3 spec was recently out.
