Distributed Systems

What distributed systems theory should a distributed systems engineer know?

• Distributed systems theory for the distributed systems engineer
• Scalable Web Architecture and Distributed Systems
• What We Talk about When We Talk about Distributed Systems (video/presentation) – from a core developer of RabbitMQ
• Design Patterns for Container-based Distributed Systems - Brendan Burns and David Oppenheimer, Google

Research Papers

• @bentanweihao / distributed-systems-references

Reactive Programming

• Reactive Manifesto
• Reactive Streams

Scalable Web Architecture

Data Pipeline

• Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)
• The Log: What every software engineer should know about real-time data’s unifying abstraction

A system that assumes an external log is present allows the individual systems to relinquish a lot of their own complexity and rely on the shared log. Here are the things I think a log can do:

• Handle data consistency (whether eventual or immediate) by sequencing concurrent updates to nodes
• Provide data replication between nodes
• Provide “commit” semantics to the writer (i.e. acknowledging only when your write guaranteed not to be lost)
• Provide the external data subscription feed from the system
• Provide the capability to restore failed replicas that lost their data or bootstrap new replicas
• Handle rebalancing of data between nodes.

• Making Sense of Stream Processing
• Questioning the Lambda Architecture
• Netflix’s Data Pipeline with Steven Wu [podcast]
• Designing for Massive Scalability at Backtype [slide deck]

Resilient services

• Building Fast and Resilient Web Applications [presentation video + short article]