I’m researching a project where I need to talk to a lot of embedded devices (let’s say it’s 10s of thousands of devices). We need to be able to send/receive commands, state, configurations, see online/offline status etc.
We are currently discussing two options that I’d like some advice on, if you have experience.
Here all devices connect to the MQTT Broker, and also the Elixir app. Other services can all the HTTP endpoints to send commands or query devices without having to deal with MQTT.
Pros:
Devices does not drop connection on deploys etc of the Elixir App
MQTT is a battle tested and performant protocol with good brokers around
Built in QoS (exactly-once, at-least-once delivery etc)
Elixir app would be a bit simpler
Cons:
Another service to configure and maintain
Expensive to pay for hosted/managed options (especially if you want persisted messages etc)
Option 2: Pure Elixir
Here we’d skip MQTT and just connect directly, like:
The idea here is that we skip MQTT and all devices connect directly to the Elixir app (maybe using Phoenix Channels if possible). We’d run processes for each device which handles and routes commands and requests appropriately.
Pros:
Only one service to maintain
Can do whatever we want in Elixir compared to only features available in the MQTT broker
Cons:
All devices will drop their connection on deploys/restarts and will have to reconnect
A more complex Elixir application
For me, I’m leaning more towards option 1 since I see a lot of value in not having to deal with the long running connections, they are instead managed by the broker. But, I also haven’t really used MQTT before (other in some toy projects)…
Has anyone done anything similar and have some advice?
I’m an Elixir beginner but also soon getting my hands dirty with IoT for my second Elixir project.
My project differs slightly from yours, in that I’m using a plain mobile device as a proxy in between the actual IoT device and my servers.
I’m starting out with plain HTTP (direct to my Elixir app) for the first HW versions (first 10-100 devices) but once that’s out of the way I had also considered the MQTT broker approach you described.
Depending on the commercial needs/availability needs of your project, but also considering you mentioned it’s a research project, would plain hosting a Mosquitto/RMQ broker with backups (say on Hetzner for example) do the trick for you?
Additionally you could mount the persistency layer directory/files (e.g. /var/lib/mosquitto/mosquitto.db) to another filesystem/disk in case things go bad on that VM/container.
This would be just a bit more work, but will likely get you an order of magnitude savings VS hosted options.
In case it feels like too much work, I for example currently have a setup on Terraform+Kamal for my infra that can be easily extended to accomodate my example above.
This of course depends on how comfortable you are around running and configuring your infrastructure - but I believe it’s a worthwhile investment.
This is what I’m doing for my initial version (but over HTTP, not websocket) but keeping the internal data units/schema consistent such that then the transition to proper network-efficient protocols is smoother.
(If you have already for sure tens of thousands of devices initially at your disposal then I’d recommend going with your first option right out of the gate - I will also go the MQTT route in case my stuff sells )
Your Option 2 is also in the danger of losing messages. I too dislike having too many apps but having a separate message broker is basically normal table stakes (and common sense) as much as having a dedicated DB is.
Centralizing message acceptors in Elixir would also mandate you having several copies of the app and gradual termination so no messages are lost. Needless complexity IMO.
So Option 1 I’d say. Plus you don’t have to choose between Kafka and RabbitMQ, for example. I’d use NATS. Looks like a better version of Kafka for anyone who doesn’t need literal hundreds of thousands of messages per second.
)
