Scalable Web Architecture and Distributed Systems - General Discussion, Blog Posts, Wiki

Distributed Systems

What distributed systems theory should a distributed systems engineer know?

Research Papers

Reactive Programming

Scalable Web Architecture

Data Pipeline

A system that assumes an external log is present allows the individual systems to relinquish a lot of their own complexity and rely on the shared log. Here are the things I think a log can do:

  • Handle data consistency (whether eventual or immediate) by sequencing concurrent updates to nodes
  • Provide data replication between nodes
  • Provide “commit” semantics to the writer (i.e. acknowledging only when your write guaranteed not to be lost)
  • Provide the external data subscription feed from the system
  • Provide the capability to restore failed replicas that lost their data or bootstrap new replicas
  • Handle rebalancing of data between nodes.

Resilient services

  • reactivemanifesto
  • reactive streams -> this is something related to Java VM but the main idea is the you have producer and consumer over network with back pressure (the speed can adapt during time to producer and consumer). This help eliminate big buffer or loosing data if buffer is full -> Assume you could add something like this to phoenix channel :slight_smile:

####Key Requirements for Streaming Platforms: A Micro-Services Advantage - Whiteboard Walkthrough