Distributed Systems
What distributed systems theory should a distributed systems engineer know?
- Distributed systems theory for the distributed systems engineer
- Scalable Web Architecture and Distributed Systems
- What We Talk about When We Talk about Distributed Systems (video/presentation) – from a core developer of RabbitMQ
- Design Patterns for Container-based Distributed Systems - Brendan Burns and David Oppenheimer, Google
Research Papers
Reactive Programming
Scalable Web Architecture
Data Pipeline
- Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)
- The Log: What every software engineer should know about real-time data’s unifying abstraction
A system that assumes an external log is present allows the individual systems to relinquish a lot of their own complexity and rely on the shared log. Here are the things I think a log can do:
- Handle data consistency (whether eventual or immediate) by sequencing concurrent updates to nodes
- Provide data replication between nodes
- Provide “commit” semantics to the writer (i.e. acknowledging only when your write guaranteed not to be lost)
- Provide the external data subscription feed from the system
- Provide the capability to restore failed replicas that lost their data or bootstrap new replicas
- Handle rebalancing of data between nodes.
- Making Sense of Stream Processing
- Questioning the Lambda Architecture
- Netflix’s Data Pipeline with Steven Wu [podcast]
- Designing for Massive Scalability at Backtype [slide deck]