Hello lovely people of the Elixir community!

I’m working on a system to monitor Nerves devices to ensure they maintain constant internet connectivity. I’d love to get your thoughts on the architecture I’m considering.

The Problem

I need to:

• Monitor a very high number of Nerves devices (potentially thousands or tens of thousands)
• Ensure each device maintains constant internet connectivity
• Use WebSocket connections with periodic ping messages to detect connectivity issues
• Handle the scale efficiently without overwhelming the system
• Ensuring that none of the messages were lost is paramount.

Proposed Architecture

I’m thinking of using Kafka as a queueing buffer to decouple the WebSocket connection handling from the actual processing and storage. This would allow for a more robust and flexible architecture that can handle bursts and scale horizontally.

Here’s the architecture I’m envisioning:

┌─────────────┐
│ Nerves      │
│ Device      │
└──────┬──────┘
       │ WebSocket Connection
       │ (ping messages)
       ▼
┌─────────────────────┐
│ Phoenix             │
│ WebSocket Handler   │
│ (Phoenix Channels)  │
└──────┬──────────────┘
       │ publish_heartbeat()
       ▼
┌─────────────────────┐
│  Kafka Producer     │
│  (kafka_ex)         │
└──────┬──────────────┘
       │
       ▼
┌─────────────────────┐
│   Kafka Topic       │
│   (device_heartbeat)│
│   ┌───────────────┐ │
│   │  Partition 0  │ │
│   │  Partition 1  │ │
│   │  Partition N  │ │
│   └───────────────┘ │
└──────┬──────────────┘
       │
       ▼
┌─────────────────────┐
│ Broadway Consumer   │
│ (broadway_kafka)    │
│  - Processors       │
│  - Batchers         │
└──────┬──────────────┘
       │ batch_insert()
       │ detect_offline()
       ▼
┌─────────────────────┐
│   PostgreSQL        │
│   (device_status)   │
└─────────────────────┘

The Flow

1. Nerves devices establish persistent WebSocket connections to a Phoenix application
2. Devices send periodic ping/heartbeat messages (e.g., every 30 seconds)
3. Phoenix Channels receive these messages and immediately publish them to Kafka (async, non-blocking)
4. Kafka acts as a buffer, handling bursts and providing durability
5. Broadway consumers process heartbeats in batches, updating device status in PostgreSQL
6. Missing heartbeats can be detected by checking timestamps in the database

Questions for the Community

1. Is Kafka overkill for this use case? Given that I’m dealing with potentially tens of thousands of devices, I thought Kafka would help with:

   • Handling bursts of heartbeat messages (f.e. after blackout reconnecting rush)
   • Decoupling WebSocket handling from processing
   • Horizontal scaling of consumers
   • Durability and replay capabilities

   But I’m wondering if there’s a simpler approach that would work just as well?
   Just Oban?

2. WebSocket connection management: For this scale, should I:

   • Use Phoenix Channels with a single topic per device?
   • Use a single topic with device IDs in the message payload?
   • Consider connection pooling or other patterns?

3. Partitioning strategy: If I go with Kafka, how should I partition the device_heartbeat topic?

   • By device ID (ensures ordering per device)?
   • Event type?
   • Round-robin (better load distribution)?
   • Other ideas?

4. Offline detection: What’s the best approach for detecting when a device goes offline?

   • Timeout-based checks in the consumer?
   • Prescence lifecycle?
   • Use Kafka’s consumer lag or other mechanisms?

5. Alternative architectures: Are there other patterns or libraries in the Elixir ecosystem that would be better suited for this use case? I’ve seen mentions of:

   • Direct database writes from Phoenix (simpler but less resilient)
   • GenStage/Flow for processing (lighter than Kafka)
   • “Just” Oban?
   • Other message queue systems

6. Performance considerations: For this scale, what should I be most concerned about?

   • WebSocket connection limits per Phoenix node?
   • Kafka producer throughput?
   • Database write performance?
   • Memory usage for connection state?

Context

I already have Kafka infrastructure set up in my project (using kafka_ex and broadway_kafka), so adding this wouldn’t require new infrastructure. However, I’m open to simpler solutions if they’re more appropriate.

Thank you for your input​