Systems · Architecture · Algorithms
I write about distributed systems, AWS architecture, networking, algorithms, and the engineering decisions hidden beneath working software.
From failure modes and trust boundaries to invariants and algorithmic patterns—each article starts with a question worth understanding deeply.
Explore the writingLatest writing
Independent timeouts across microservices can outlive the user request they were meant to serve. This article explains end-to-end deadlines, remaining-budget propagation, cancellation, retries, queue time, local safety caps, gRPC deadline semantics, partial results, idempotency, and how deadline-aware systems avoid wasting capacity on work whose result is already useless.
Find a note
18 published articles
Browse by topic
Recent notes
databases
Zero-downtime schema evolution is a compatibility problem, not just a migration command. This article walks through expand-and-contract migrations, dual writes, resumable backfills, validation, feature-flagged cutovers, rollback windows, ORM hazards, online DDL, and the operational discipline required when old and new application versions share one production database.
reliability-engineering
A redundant link can fail over perfectly while the system still degrades. This article examines failure-state capacity, shared failure domains, packet-processing limits, Suricata, RSS, queues, interrupt distribution, and how to benchmark the bottleneck that appears after redundancy does its job.
cloud-infrastructure
A Kubernetes Deployment can report a successful rollout while users still see 502s, 503s, or connection resets. This article explains the gap between Pod availability and request completion, including EndpointSlice termination state, load-balancer propagation, graceful shutdown, preStop hooks, health probes, rollout capacity, long-lived connections, PodDisruptionBudgets, and how to test deployments with real in-flight traffic.
distributed-systems
Updating a database and publishing an event looks like one operation in application code, but PostgreSQL and Kafka do not share a transaction. This article explains the dual-write problem, transactional outbox and inbox patterns, idempotent consumers, ordering, CDC with Debezium, reconciliation, sagas, event versioning, and how to make cross-system failure recoverable instead of ambiguous.
security
Authentication can prove who a user is while a multi-tenant application still exposes the wrong customer’s data. This article examines tenant context, scoped queries, caches, jobs, stale claims, privileged access, and the architectural boundaries that make isolation enforceable.
cloud-infrastructure
Serverless compute can scale faster than databases, caches, queues, and third-party services behind it. This article examines safe concurrency, database connection limits, RDS Proxy, reserved concurrency, SQS backpressure, throttling, cache failures, load testing, and how to keep elastic compute from overwhelming finite dependencies.
reliability-engineering
A distributed system can look healthy at the component level while the user-visible workflow is already failing. This article examines tail latency, semantic failures, end-to-end counts, Kafka lag, tenant slices, retries, queues, black-box monitoring, traces, business telemetry, alert design, and how to build dashboards around the guarantees the system is actually supposed to provide.
distributed-systems
Reconnection in a local-first collaborative system means more than restoring a socket. It requires preserving offline work, merging replicas, rebuilding presence, revalidating access, and knowing when the application can honestly say it is synced.
performance-engineering
Increasing threads can improve a Java service until CPU, database connections, locks, queues, memory, or downstream dependencies become the real bottleneck. This article examines Little’s Law, thread pools, HikariCP, queueing, contention, garbage collection, virtual threads, bulkheads, backpressure, and how to find the concurrency level that produces useful throughput instead of overload.
distributed-systems
Kafka consumer lag is a symptom, not a diagnosis. This article explains how to reason about backlog through arrival rate, processing rate, event age, partition skew, downstream bottlenecks, poll timing, poison records, rebalances, and autoscaling—and why adding consumers can sometimes increase lag instead of reducing it.
cloud-infrastructure
Why a successful S3 deployment can still look stale behind CloudFront, how invalidation fixes it, and how to automate that step with GitHub Actions.
distributed-systems
A cache can begin as a latency optimization and quietly become essential to keeping the backing system alive. This article examines cache stampedes, TTL jitter, stale serving, invalidation races, negative caching, hot keys, eviction, load shedding, cache-format compatibility, and how to reason about the system when the hit rate suddenly becomes zero.
databases
A Cassandra schema can perform beautifully until workload skew, hot and wide partitions, tombstones, compaction debt, repair pressure, and JVM pauses accumulate in the tail. This article examines how partition design ages in production, why averages and extra nodes can hide the problem, and how to design and test Cassandra around the worst partition rather than the typical one.
distributed-systems
Distributed rate limiting is not just a counter problem. This article examines Redis-backed limits, token buckets, concurrency controls, fail-open and fail-closed behavior, local fallbacks, regional budgets, replica staleness, hot keys, approximation error, load shedding, and how to choose the right consistency policy for the resource being protected.
distributed-systems
Retries can hide transient failures, but every additional attempt also creates load. This article examines failure classification, idempotency, deadlines, exponential backoff, jitter, retry budgets, load shedding, layered amplification, and how to decide when the reliable choice is to stop retrying.
distributed-systems
Kafka can provide exactly-once processing within a well-defined transactional boundary, but databases, payment APIs, email, files, and other external side effects sit outside it. This article examines idempotent producers, Kafka transactions, read_committed consumers, producer fencing, database deduplication, business idempotency, replay, deterministic processing, and how to design correctness across system boundaries.
distributed-systems
Building reliable webhook consumers that stay correct through retries, duplicates, out-of-order delivery, partial failures, and reconciliation.
Have a question about something I wrote?
Contact