Skip to content
SkillPilotNavigate Your Career

Observability Engineer Roadmap

A path into observability as a craft of its own — wide events, signal correlation, telemetry cost, collector pipelines, high-cardinality analysis, continuous profiling, and running observability as a platform other teams consume.

Entry level
$135,441
Observability Engineer total pay estimate, 25th percentile (US, 27 salaries submitted)
Mid level
$164,046
Observability Engineer total pay estimate (US average, 27 salaries submitted)
Senior level
$202,686
Observability Engineer total pay estimate, 75th percentile (US, 27 salaries submitted)
Glassdoor

This path assumes you have shipped services and used a metrics or logging tool, but have never owned the telemetry itself. It is a craft distinct from SRE: an SRE decides what reliability to target, an observability engineer builds the system that can answer why it missed.

The premise, and the reason the ordering looks unusual: you cannot ask a question of data you already aggregated away. A metric with the cardinality stripped out cannot tell you which customer is affected. So wide events come early, and dashboards do not come at all — they answer questions you already knew to ask.

Cost is phase five rather than an appendix, because telemetry is the one system whose bill scales with how well it works.

Expect 6–8 months. Phase seven is where the mental model clicks or does not.

New to Linux and the command line?

This path assumes fundamentals you may not have yet. Our Foundations Pack is out and free — Linux, the shell and Git, with exercises that mark your work and explain why you got it wrong. We're writing an agents pack next; leave your email if you want to hear when it ships.

One email when the pack launches. No spam, unsubscribe any time.

Your progress0%

The path, phase by phase

  1. Observability vs Monitoring

    Monitoring answers questions you wrote down in advance. Observability is the property that lets you answer one nobody anticipated, without shipping new code first. Done when you take a live service, ask three questions its dashboards were never built for, and record which ones its telemetry can already answer. The unanswerable ones define the rest of this roadmap.

    2-3 weeks
    7 Skills
    Observability vs Monitoring
    Cardinality & Dimensionality
    Unknown-unknowns Debugging
    Signal Taxonomy
    Telemetry Data Models
    Question-driven Design
    Debugging from First Principles
    Show details, projects and resources

    Skills you'll master

    Observability vs Monitoringintermediate
    Cardinality & Dimensionalityadvanced
    Unknown-unknowns Debuggingadvanced
    Signal Taxonomyintermediate
    Telemetry Data Modelsintermediate
    Question-driven Designadvanced
    Debugging from First Principlesadvanced

    Hands-on projects

    1. 01Take a service you did not build, write down three questions its dashboards cannot answer, and classify each as missing signal, missing dimension or missing correlation
    2. 02Reproduce a bug that only affects one customer, and record every point where you had to add code or SSH somewhere to make progress
    3. 03Write the one-page distinction between monitoring and observability you would give a sceptical engineering manager, without using the word platform
    4. 04Take one existing dashboard and list what it would take to slice every panel by customer, region and build version — the cost estimate is the deliverable
    5. 05Catalogue every telemetry signal a single service emits today, with its data model, its owner and whether anyone queried it in the last quarter
    6. 06Find a past incident whose diagnosis needed a code deploy to add a log line, and write what dimension would have made the deploy unnecessary

  2. Wide Events as the Unit of Telemetry

    The craft's central idea: emit one wide, richly dimensioned event per unit of work instead of scattering the story across counters and log lines. Done when a single service emits events carrying at least thirty attributes each, and you can answer a new question by filtering on a dimension rather than by deploying. Pre-aggregated metrics discard exactly the outliers you will be asked about.

    4-5 weeks
    8 Skills
    Wide Event Design
    Attribute Modelling
    OpenTelemetry SDKs
    Semantic Conventions
    Structured Logging
    Auto vs Manual Instrumentation
    Instrumentation Libraries
    Aggregation Loss
    Show details, projects and resources

    Skills you'll master

    Wide Event Designadvanced
    Attribute Modellingadvanced
    OpenTelemetry SDKsadvanced
    Semantic Conventionsadvanced
    Structured Loggingadvanced
    Auto vs Manual Instrumentationadvanced
    Instrumentation Librariesintermediate
    Aggregation Lossadvanced

    Hands-on projects

    1. 01Instrument one service to emit a single wide event per request carrying build version, customer tier, region, feature flags and downstream timings
    2. 02Take a question your metrics cannot answer, answer it from wide events instead, and write down which aggregation had destroyed the information
    3. 03Adopt OpenTelemetry semantic conventions across a service and list every custom attribute you kept, with the reason each one is not covered by the spec
    4. 04Add zero-code auto-instrumentation to a service, then measure what it captured and what it missed against a hand-instrumented path
    5. 05Write and enforce the attribute naming schema for a codebase, including which attributes are forbidden because they are unbounded identifiers
    6. 06Convert one service's plaintext logs into structured events joined to their trace context, then delete every log line the new events made redundant

  3. Context Propagation Across Boundaries

    Wide events only compose into a story if they share a trace context, and context is what breaks first at every boundary you did not write. Done when one request keeps its trace context across an async queue, a thread pool and a third-party SDK, and you can name the exact library that dropped it when it broke. Baggage is how a dimension known at the edge reaches a service ten hops down.

    3-4 weeks
    8 Skills
    Context Propagation
    W3C Trace Context
    Baggage Propagation
    Async & Queue Boundaries
    Span Design & Nesting
    Cross-language Interop
    Broken Trace Debugging
    OTLP Protocol
    Show details, projects and resources

    Skills you'll master

    Context Propagationadvanced
    W3C Trace Contextadvanced
    Baggage Propagationadvanced
    Async & Queue Boundariesadvanced
    Span Design & Nestingadvanced
    Cross-language Interopintermediate
    Broken Trace Debuggingadvanced
    OTLP Protocolintermediate

    Hands-on projects

    1. 01Propagate trace context through a message queue so a producer span and its consumer span appear in one trace, and document the header you had to carry by hand
    2. 02Break context propagation deliberately in a thread pool, observe the orphaned spans, then fix it and write the diagnostic that identified the gap
    3. 03Use baggage to carry a customer tier attribute from the edge gateway to a service four hops downstream, and query events by it at the far end
    4. 04Trace one request across two services written in different languages, and record every place the two SDKs disagreed about span naming
    5. 05Audit a service's span hierarchy and cut the spans that carry no decision value — depth without information makes traces slower to read, not richer
    6. 06Instrument a call through a third-party client library that does not propagate context, and write the wrapper that makes it behave

  4. Signal Correlation

    Three tools holding three signals is three tools, not observability. This phase is the joins: exemplars from a metric into a trace, a trace into its events, a profile into the span that produced it. Done when a responder moves from a latency spike to the exact slow trace to its logs without copying an identifier by hand, and you can name every field that makes each hop possible.

    3-4 weeks
    7 Skills
    Metric-to-Trace Exemplars
    Trace-to-Log Correlation
    Resource Attributes
    Identifier Consistency
    Grafana Data Source Links
    Exploratory Workflows
    Investigation Ergonomics
    Show details, projects and resources

    Skills you'll master

    Metric-to-Trace Exemplarsadvanced
    Trace-to-Log Correlationadvanced
    Resource Attributesadvanced
    Identifier Consistencyadvanced
    Grafana Data Source Linksintermediate
    Exploratory Workflowsadvanced
    Investigation Ergonomicsadvanced

    Hands-on projects

    1. 01Wire exemplars from a histogram metric into traces so clicking a p99 bucket lands on a real slow request
    2. 02Configure trace-to-log correlation so every span links to the log lines it emitted, then time a responder doing the hop with and without it
    3. 03Standardise resource attributes — service, version, environment, instance — across every signal, and find the service that was silently naming itself two different things
    4. 04Run a timed investigation drill: from an alert to a named root cause, counting every manual copy-paste of an identifier as a defect to fix
    5. 05Build the single exploration view an on-call responder starts from, then hand it to someone unfamiliar and watch where they get stuck
    6. 06Correlate a deployment marker with a latency change, so a responder can see which build introduced the regression without asking anyone

  5. Telemetry Cost as a Design Constraint

    Every attribute you add has a price, and observability budgets are cut by finance, not by engineers. This phase makes cost a first-class input rather than a quarterly surprise. Done when you can state the cost per million events for your own pipeline, attribute the bill to the services generating it, and cut it by a third while naming exactly which question you gave up answering.

    2-3 weeks
    7 Skills
    Telemetry Cost Modelling
    Cost per Signal Attribution
    Retention Tiering
    Cardinality Budgets
    Storage & Ingest Economics
    Vendor Pricing Models
    Value-per-Byte Analysis
    Show details, projects and resources

    Skills you'll master

    Telemetry Cost Modellingadvanced
    Cost per Signal Attributionadvanced
    Retention Tieringadvanced
    Cardinality Budgetsadvanced
    Storage & Ingest Economicsadvanced
    Vendor Pricing Modelsintermediate
    Value-per-Byte Analysisadvanced

    Hands-on projects

    1. 01Calculate the cost per million events for your own telemetry pipeline, broken down into ingest, storage and query
    2. 02Attribute a month of telemetry spend to the services that generated it, and take the top spender its own number
    3. 03Cut telemetry cost by a third and document the exact question you can no longer answer — an unnamed loss means you did not measure the tradeoff
    4. 04Set a cardinality budget per service, enforce it, and record what the first team to breach it was actually trying to do
    5. 05Implement tiered retention so recent data is queryable and older data is cheap, then test that a three-month-old investigation is still possible
    6. 06Model the same telemetry volume under two vendors' pricing and write the one-page comparison, including what each pricing model quietly incentivises

  6. Sampling Strategies

    Sampling is how you keep the interesting telemetry and drop the rest, and getting it wrong deletes the evidence for the incident you have not had yet. Done when you run tail-based sampling in production and can prove, against a held-out full-fidelity window, that every slow and every failed request survived. Head sampling is cheap and decides before it knows whether the request was interesting.

    3-4 weeks
    7 Skills
    Head vs Tail Sampling
    Tail Sampling Policies
    Sampling Bias Analysis
    Statistical Reweighting
    Consistent Sampling Decisions
    Error & Latency Preservation
    Sampling Verification
    Show details, projects and resources

    Skills you'll master

    Head vs Tail Samplingadvanced
    Tail Sampling Policiesadvanced
    Sampling Bias Analysisadvanced
    Statistical Reweightingadvanced
    Consistent Sampling Decisionsadvanced
    Error & Latency Preservationadvanced
    Sampling Verificationadvanced

    Hands-on projects

    1. 01Implement tail-based sampling with separate policies for errors, slow requests and a baseline of successes, and document each policy's intent
    2. 02Run full fidelity and your sampling configuration side by side for a day, then prove no error trace and no p99 request was lost
    3. 03Reweight sampled counts back to true traffic volume and reconcile the estimate against an unsampled counter
    4. 04Find a sampling configuration that silently drops one customer's traffic entirely, and write the check that would have caught it
    5. 05Make sampling decisions consistent across services so a trace is never half kept, and demonstrate a partial trace before and after
    6. 06Compare the cost and the fidelity of head and tail sampling on the same workload, and write which you would run and under what volume you would switch

  7. Query & High-Cardinality Analysis

    Telemetry you cannot interrogate quickly is an archive. This phase is the analytical craft: slicing by high-cardinality dimensions until an anomaly resolves into a population. Done when you take an unexplained latency spike and identify the shared attribute of the affected requests through query alone, in under fifteen minutes, with no prior hypothesis about the cause.

    3-4 weeks
    8 Skills
    PromQL
    LogQL
    TraceQL
    High-Cardinality Slicing
    Outlier & Population Analysis
    Percentiles & Distributions
    Query Performance Tuning
    Hypothesis-free Investigation
    Show details, projects and resources

    Skills you'll master

    PromQLadvanced
    LogQLadvanced
    TraceQLadvanced
    High-Cardinality Slicingadvanced
    Outlier & Population Analysisadvanced
    Percentiles & Distributionsadvanced
    Query Performance Tuningintermediate
    Hypothesis-free Investigationadvanced

    Hands-on projects

    1. 01Take an unexplained latency spike and find the shared attribute of the affected requests by iterative slicing, timing yourself from alert to answer
    2. 02Write the ten PromQL, LogQL and TraceQL queries an on-call engineer actually needs, and store them where they are found under pressure rather than in a wiki
    3. 03Find a case where the median improved while the p99 degraded, and identify the population hiding behind the average
    4. 04Rewrite a query that times out on a month of data until it returns in seconds, and document which part of the data layout made the difference
    5. 05Diagnose an incident using only query, no dashboards, and record which panels you never needed — those are candidates for deletion
    6. 06Build a saved analysis that segments errors by build version and customer tier simultaneously, so a bad release is visible before anyone reports it

  8. Telemetry Pipelines & Collectors

    The layer between what applications emit and where it lands, and the thing that turns telemetry from a per-service decision into infrastructure. Done when you run a collector fleet that receives, transforms, redacts and routes telemetry to more than one backend, survives a backend outage without losing data, and reports on its own health. A pipeline you cannot observe is a blind spot on the observability team.

    4-5 weeks
    8 Skills
    OpenTelemetry Collector
    Receivers, Processors & Exporters
    Pipeline Topology Design
    Telemetry Transformation
    PII Redaction & Scrubbing
    Backpressure & Buffering
    Collector Scaling & HA
    Pipeline Self-observability
    Show details, projects and resources

    Skills you'll master

    OpenTelemetry Collectoradvanced
    Receivers, Processors & Exportersadvanced
    Pipeline Topology Designadvanced
    Telemetry Transformationadvanced
    PII Redaction & Scrubbingadvanced
    Backpressure & Bufferingadvanced
    Collector Scaling & HAadvanced
    Pipeline Self-observabilityadvanced

    Hands-on projects

    1. 01Deploy the OpenTelemetry Collector in both agent and gateway roles, and write down which processing belongs at each tier and why
    2. 02Route the same telemetry stream to two backends at once, then cut one over to a new vendor without touching a single application
    3. 03Write a transform pipeline that redacts PII from attributes before export, and verify it against a payload deliberately seeded with test identifiers
    4. 04Kill a backend under load and measure exactly how much telemetry the collector's queue and retry saved — the number is usually not what the defaults imply
    5. 05Scale a collector fleet to handle a 10x ingest burst, and document which resource saturated first
    6. 06Instrument the collector itself, so an on-call engineer can tell telemetry loss from a genuine drop in traffic

  9. Observability as a Platform Service

    The multiplier, and where this role stops being a specialism and becomes infrastructure. Done when a team you do not sit with instruments a new service to your standard on their own, using your defaults, without asking you a question. Adoption you had to chaperone is a demo. Governance that only says no produces shadow telemetry pipelines you find out about on the invoice.

    2-3 weeks
    8 Skills
    Observability as a Platform
    Golden Path Instrumentation
    OpenTelemetry Operator
    Telemetry Governance
    Self-service Onboarding
    Multi-tenancy & Isolation
    Internal Developer Enablement
    Influence Without Authority
    Show details, projects and resources

    Skills you'll master

    Observability as a Platformadvanced
    Golden Path Instrumentationadvanced
    OpenTelemetry Operatoradvanced
    Telemetry Governanceadvanced
    Self-service Onboardingadvanced
    Multi-tenancy & Isolationadvanced
    Internal Developer Enablementadvanced
    Influence Without Authorityintermediate

    Hands-on projects

    1. 01Publish an instrumentation golden path — an SDK wrapper and a default collector config — and count teams who adopt it without you in the room
    2. 02Deploy the OpenTelemetry Operator so new workloads are instrumented by annotation rather than by a code change in each repository
    3. 03Write the telemetry governance policy covering attribute naming, retention and cardinality limits, then measure how often teams request exceptions
    4. 04Run a multi-tenant backend where one team's cardinality explosion cannot degrade another team's queries, and prove it with a deliberate blast
    5. 05Onboard a new service end to end using only your documentation, with someone else driving — the questions they ask are the real backlog
    6. 06Report platform-level adoption to leadership as a small set of KPIs, each with the query behind it so nobody recalculates by hand

  10. Continuous Profiling

    The signal almost nobody has. Traces tell you which service was slow; profiles tell you which line of code burned the CPU while it was. Done when profiling runs continuously in production at acceptable overhead and you have used a flame graph to find a hot path that traces alone had only narrowed to a service. Profiling on demand finds nothing, because you enable it after the interesting minute has passed.

    2-3 weeks
    7 Skills
    Continuous Profiling
    Flame Graph Analysis
    CPU & Memory Profiling
    eBPF-based Profilers
    Grafana Pyroscope
    Profile-to-Trace Correlation
    Profiling Overhead Management
    Show details, projects and resources

    Skills you'll master

    Continuous Profilingadvanced
    Flame Graph Analysisadvanced
    CPU & Memory Profilingadvanced
    eBPF-based Profilersadvanced
    Grafana Pyroscopeadvanced
    Profile-to-Trace Correlationadvanced
    Profiling Overhead Managementadvanced

    Hands-on projects

    1. 01Run continuous CPU profiling in production for a week and measure the actual overhead against the claimed figure
    2. 02Use a flame graph to find a hot path traces had only narrowed to a service, and quantify the CPU reclaimed after fixing it
    3. 03Correlate a profile with the span that produced it, so a slow trace links to the function that made it slow
    4. 04Track allocation profiles across a release and identify a memory regression before it became an out-of-memory incident
    5. 05Deploy an eBPF-based profiler that requires no application changes, and compare its resolution against in-process profiling on the same workload
    6. 06Write the guidance a product team needs to read a flame graph without you — then test it by asking one to diagnose a seeded regression

What the job is actually like

Day to day
Two jobs sharing one title, and which you have depends on the company. In the platform version you run the telemetry infrastructure other teams depend on, and the week is collector pipelines, cardinality that exploded overnight, and a bill growing faster than the traffic. In the practitioner version you are embedded with product teams, helping them instrument things properly and answering questions nobody could answer last month. Both involve far more cost conversation than anyone expects — the interesting constraint in this field is economic, not technical. Ask which version the job is before you accept it.
The interview
Instrumentation is where the round is won: given this service, what would you record, and the good answer is about the questions you want to answer later rather than a list of metrics. Expect a debugging exercise using telemetry alone rather than access to the machine, and a cost question — this data is expensive, halve it without losing the ability to investigate. Cardinality comes up specifically and often, because it separates people who have run these systems from people who have used them. Vendor experience matters less than teams assume, but OpenTelemetry is now assumed knowledge rather than a bonus.
How people get in
Mostly from reliability and infrastructure work rather than through a dedicated pipeline. Site reliability engineers arrive already fluent in what they wish they could see and specialise into building it — this roadmap is that specialisation, and the two paths share their instrumentation phases. Backend engineers arrive from the other direction, having instrumented their own services well enough to want it as the whole job. Data engineers occasionally arrive and are unusually strong on the pipeline half. What does not transfer is a monitoring mindset in which the dashboards are the deliverable.
After senior
The platform fork widens until observability is one of several internal products you own, which is the platform engineering path under another name. A second fork stays technical and deepens into telemetry infrastructure at a scale where the pipeline is itself a distributed systems problem. A third goes to the vendors, who hire heavily from practitioners and pay well for people who have run the thing at scale. The development worth planning for is that model and agent behaviour is now being treated as an observability problem, opening a specialisation that did not exist two years ago.
Why people leave
The dashboard trap is the first: beautiful surfaces nobody opens during an incident, because they answer questions nobody was asking. The measure is whether the telemetry shortened an investigation, and it is worth demanding that evidence about your own work. The second is being handed the bill without the authority — told to halve telemetry spend while having no say over what teams emit, which is a negotiation dressed as an engineering problem. The third is tool ownership as the entire job, where the role decays into administering a vendor product and the analysis belongs to somebody else.

Frequently asked questions

Related certifications

Related roadmaps