Hey guys. I need a solution for c2/data aquisition& analysis server to have a persistent connection to agents in up to 100K agents per server range. Ideally with low overhead, multiplexing and ideally some QoS to prioritize control channel messages over the data channel. Any good recommendations/advice for Elixir or Erlang lib.(s) that could help would be really appreciated (as well as arch. tips or any other relevant wisdom 
) . The aim is to build a POC that will potentially convince people to adopt Elixir for at least communication/c2 portion of the product.
1 Like
/me has no clue what c2 is
So basically 100k TCP connections or something? Or usermode SCTP?
1 Like
yep up to 100K TCP
1 Like
In that caseā¦
QoS would likely be handled at the hardware side, multiplexing maybe too unless you go SCTP or so, as for low overhead, youād probably want UDP instead if you can tolerate loss (unless TCP/SCTP is low enough overhead already)?
1 Like
TCP is fine just donāt want to roll my own solution for multiplexing control and data āchannelsā with ability to prioritize control channel over data channel (and ideally ability to throttle data channel). Although it might not be too much work? Havenāt really done any network servers since college pretty much was doing purely web dev. work.
1 Like
Yeah thatās not really a TCP feature though, more an SCTP feature (and sadly not a usermode SCTP thing last I saw, though usermode SCTP can do the rest of it quite well)ā¦ ^.^;
Performing it on the software side can still cause the TCP buffer to grow in the kernel to the window size, which then causes the throughput to tank when it does start back up for a while. If you really want to throttle honestly Iād use something more like UDP, which will scale much better, and using usermode SCTP (emulated on UDP instead of being IP level SCTP since a lot of routers and Windows out in the wild are retardedly stupid for not following specs properly, which is why usermode SCTP exists at all instead of just using the IP level SCTP protocol) would gain you a lot of functionality essentially āfor freeā especially if you want reliability (if you donāt want reliability, then honestly just use UDP straight as the hardware stack usually throttles it automatically to keep the network useable, āin generalā).
Overall though, need more information about the setup, the information, how reliable it needs to be, how ordered it needs to be, how large it will be at most, how sporadic it will be, is it stateful or stateless packets, etcā¦ etcā¦
2 Likes
SCTP looks interesting. In the current product there is no persistent connection between server and agents agent checks in periodically with the server and might receive some jobs to run or commands to execute and also provides info on itās state, status of jobs etc. Some job results could be large files (few gigs) but mostly itās low volume. There are downsides to this model (mainly having to wait for checking interval) so my idea is to build a POC that has agent establish persistent connection to the server. It might make sense to have persistent SCTP connection for command and control and have an agent establish TCP connection when it needs to move large amount of data I guess.
1 Like
SCTP is entirely suited for large data as well, even more so than TCP due to itās multiplexing and multiple channels (you can still send other data on one channel even while sending huge data on another channel for example).
1 Like
Cool thank you just started reading up on SCTP got confused by message oriented vs stream oriented thing. Looks like it totally could be the solution thank you for all the info will continue to dig in
1 Like
Honestly SCTP would have taken over instead of TCP if not for Windows being monumentally stupid like always. 
1 Like
Also, for IP level SCTP, the BEAM comes with it built in. For usermode SCTP (built on UDP) I think it might be in already, either that or it still has a PR for itā¦
EDIT1: Looks like itās still pending as a PR, so youād need to implement it yourself (or swipe this implementation), maybe you can push it along to get it merged? 
EDIT2: Or of course just use UDP and implement whatever reliability you want manually (maybe a side TCP channel too, in which case can just use TCP entirely if you are okay with the limitations). ^.^:
/me may be getting too detailed, TCP is fine for most usesā¦
2 Likes
