Site Reliability Engineer Roadmap

Skills you'll master

Hands-on projects

1. 01Define SLIs and SLOs for a checkout API, including a written error-budget policy stating exactly what stops when the budget is exhausted and who decides
2. 02Take a service you did not build and write its SLO from the outside, then defend the number against both "why not higher?" and "why not lower?"
3. 03Calculate the permitted downtime for 99%, 99.9%, 99.95% and 99.99% and write down what each would actually cost to defend
4. 04Find a service whose current performance was quietly adopted as its target, and write the argument for what the target should be instead
5. 05Write the one-page error-budget policy you would actually take to a product manager, in language that survives contact with someone who does not care about percentiles
6. 06Pick a real outage from a public postmortem and work out what it cost in error budget — then say whether the SLO was set correctly

Resources

Some resources are affiliate links, marked AFF. They cost you nothing extra, we may earn a commission, and a free alternative is listed alongside wherever one exists.

Skills you'll master

Hands-on projects

1. 01Instrument a service with the four golden signals — latency, traffic, errors, saturation — using OpenTelemetry rather than a vendor SDK
2. 02Build a dashboard an on-call engineer can read at 3am without you, then test that claim by handing it to someone else and saying nothing
3. 03Replace an average-latency panel with a histogram-backed p50/p95/p99 view and write down which users the average was hiding
4. 04Deliberately blow up metric cardinality in a test environment, watch what it costs, and then fix it
5. 05Write the metric naming convention for a service and apply it retroactively to everything already emitting
6. 06Compute an SLO compliance figure directly from your own metrics, and reconcile it against what your provider's dashboard claims

Resources

Skills you'll master

Hands-on projects

1. 01Instrument a multi-service application so one request can be followed end to end in a single trace, including across an async queue boundary
2. 02Take an incident you have already resolved and re-derive its cause using only traces and logs — then note every step where you wanted to SSH somewhere
3. 03Convert a service's plaintext logs to structured events with consistent field names, and delete the fields nobody queried in six months
4. 04Implement tail-based sampling and prove you still captured the slow requests you cared about
5. 05Cut a telemetry bill in half without losing the ability to debug — write down what you gave up and why it was safe
6. 06Correlate a trace, its logs and its metrics for a single failing request, so a responder can move between all three without copying identifiers by hand

Resources

Skills you'll master

Hands-on projects

1. 01Replace a static threshold alert with a multi-window burn-rate alert and explain what each window is designed to catch
2. 02Audit an alerting setup and cut false positives by at least half, proving no real incident would have been missed
3. 03Review three months of pages: how many were actionable, and what would you delete?
4. 04Write a runbook for one service, then have someone unfamiliar with it follow the runbook during a simulated incident — the gaps they hit are the real output
5. 05Write an on-call rotation policy covering escalation, handoff and burnout rules for a four-person team
6. 06Convert a cause-based alert ("disk is 80% full") into a symptom-based one ("users are seeing errors") and argue which you would rather be woken by

Resources

Skills you'll master

Hands-on projects

1. 01Track where your operational time actually goes for two weeks, then rank the toil by hours before automating anything
2. 02Write a self-healing automation for one specific failure mode, with audit logging and a documented way to disable it in a hurry
3. 03Run a controlled chaos experiment against staging — terminate a pod under load — and document blast radius and recovery time
4. 04Automate the top item from your toil ranking, then measure the hours actually reclaimed and compare against what you predicted
5. 05Write the failure mode analysis for your own automation, answering what happens when it misfires at 3am
6. 06Build a canary deployment that automatically rolls back on an SLO burn-rate signal rather than on a human noticing

Resources

Skills you'll master

Hands-on projects

1. 01Run a tabletop simulation with assigned incident commander and scribe roles, and hold the commander to coordinating rather than debugging
2. 02Define severity levels for your service, with a concrete example of each, so classification does not get argued about during an incident
3. 03Write the stakeholder update template for a live incident — the one that says what is broken and what to expect next without promising a fix time
4. 04Practise mitigating before diagnosing: roll back, drain, or fail over first, and note how much faster users recovered than root cause was found
5. 05Run an incident handoff across a shift boundary and document what the incoming commander needed that was not written down

Resources

Skills you'll master

Hands-on projects

1. 01Produce a full postmortem within 48 hours of a simulated incident, with owners and dates on every action item
2. 02Rewrite a blameful postmortem into a blameless one and note exactly what changed in the findings, not just the tone
3. 03Go back to postmortems from six months ago and check which action items were actually completed — the completion rate is the real health metric
4. 04Analyse a quarter of incidents for patterns: which contributing factor shows up repeatedly, and what would fix the class rather than the instance?
5. 05Write the public-facing version of an incident report for customers, saying what happened without either hiding it or drowning them in internals
6. 06Run a postmortem review meeting where the author is not the most senior person in the room, and note what that changed

Resources

Skills you'll master

Hands-on projects

1. 01Restore a production-sized database from backups into a clean environment and record the real recovery time, not the estimated one
2. 02Run a schema migration on a live table with no downtime, using expand-and-contract, and document the rollback path at each step
3. 03Trigger a database failover under write load and measure exactly how much data and how many seconds were lost
4. 04Define RTO and RPO for one stateful service, then test whether your current setup actually meets them — most do not
5. 05Detect a deliberately introduced data corruption using checksums or reconciliation, and write the runbook for how it would be caught in production
6. 06Exhaust a connection pool under load and document what the application did — the failure mode is rarely the one people expect

Resources

Some resources are affiliate links, marked AFF. They cost you nothing extra, we may earn a commission, and a free alternative is listed alongside wherever one exists.

Skills you'll master

Hands-on projects

1. 01Run a capacity exercise forecasting infrastructure for 3x traffic growth over six months, showing the arithmetic rather than a guess
2. 02Load-test a service to the point of failure and record which resource ran out first — it is rarely the one you sized for
3. 03Implement load shedding so the service degrades predictably under overload instead of collapsing, and prove it with a load test
4. 04Cut infrastructure spend by 20% on one service without changing its SLO, and document what headroom you gave up
5. 05Right-size Kubernetes requests and limits from observed usage rather than from the defaults someone copied, and measure the reclaimed capacity
6. 06Tune an autoscaler until it responds to a real traffic spike without flapping — then write down the tradeoff you settled on

Resources

Skills you'll master

Hands-on projects

1. 01Design a multi-region failover for a stateful service with RTO and RPO targets, then actually test the failover rather than only documenting it
2. 02Map dependencies across a five-plus service system, identify the single points of failure, and fix the highest-risk one
3. 03Write a production readiness review checklist and run it against a service that is already live — what it fails is the point
4. 04Build a reliability capability other teams consume as a service, such as a default dashboard or SLO template they get for free
5. 05Implement graceful degradation somewhere, so the system sheds a non-critical feature under load instead of failing whole
6. 06Add circuit breakers with timeouts and jittered retries to a service-to-service call, then prove the retry storm you prevented was real

Resources