A
Anish Yennapusa
Guest
Working with microservices introduces an inherent ephemerality. Subcomponents change state and influence behavior frequently based on various conditions of the environment the system runs in. Incidents in production rarely happen against a stationary architecture.
A deployment can replace pods, a scheduler can move workloads, endpoints change, and service-to-service traffic may shift while an incident is still developing. The state of the service that an investigating engineer sees after pulling up dashboards may already differ from the state that produced the first symptoms. Any modern observability system can provide metrics, traces, and change logs for a root cause analysis. The analytical problem faced by the investigator is not whether there is sufficient historical context or visibility but instead
understanding what changes took place between the current degraded state and the last known healthy state of the system.
Essentially, it becomes an issue of reconstruction: whether we can reconstruct enough of the incident-time topology to understand what changed around the failure and then correlate those structural changes with behavioral evidence.
A conventional dependency graph to capture architecture relationships can be represented as G = (V, E), where vertices represent entities and edges represent relationships. For incident analysis, the additional dimension of time models the abstraction as G(t) to capture the relationships that were valid at a specific point in time. This distinction is important to tracing how the system changed over a sequence of events. This time-aware topology model treats those connections as temporal state.
If t1 represents the last known-good state and t2 represents the beginning of degradation, the difference between G(t1) and G(t2) identifies structural changes that occurred around the incident. While the changes are not necessarily root causes, they constitute a focused set of contributors that can be evaluated using telemetry and other evidence.
Incident analysis does not need to carry a comprehensive record of each relationship of the service in the production environment. Holding onto data that will likely not produce insights will inevitably increase operational overhead; to reduce noise and be economical with service costs, it is prudent to target critical information only. The useful topology is the subset that explains the operating context of the affected service, its logical dependencies, the workload instances serving it, where those workloads were running, as well as any infrastructure relationships that could materially influence behavior.
The right set of data to look for ultimately depends on the incident. A latency change may need just a service-level dependency graph to investigate while a placement-sensitive incident may require the service to be expanded into pods and nodes. We must be cognizant of this multi-resolution approach without turning it into a high-cardinality record of every runtime interaction for every investigation.
The most value we get from reconstruction is in comparing topology states and then evaluating the changed entities against behavioral telemetry. By correlating state changes to particular behavior, we can begin to build a causal analysis, almost a frame-by-frame playback of production incidents. The following framework provides a generic breakdown of the application of this approach.
For investigation, the incident timestamp should be identified as the first observable point when service degradation began, which may not be when the underlying fault first occurred. This is not the alert timestamp, which is often the point after which a certain threshold has been breached for some time. The first indication of the behavioral change that led to the incident is a better marker, which allows us to correlate other factors that could have influenced a change in behavior.
In the representative example, Checkout latency begins increasing at 10:35 even though the alert does not fire until 10:38.
Tincident = 10:35
To measure the deviation in the degraded system, we must establish a point of comparison with a state that is known to be healthy. The correct window before the incident before selecting a stable period is highly subjective, volatile systems will need to be baselined against a longer period while relatively steady ones can be traced closer to the degradation incident.
Tgood = 10:30
Taking each of the above timestamps, we establish the relationships for the service at each period. The reconstruction should be scoped to target pertinent information and expand into auxiliary dependencies only as required. i.e application dependencies, workload instances, placement and infrastructure should be pulled in on a case-by-case basis. This reconstruction depends on having enough historical relationship data available for those timestamps; missing records can limit what we can compare.
Ggood = G(10:30)
Gincident = G(10:35)
The topology mappings provide a directed view of the service states that highlight changes in entities and relationships, which include placement transitions, replaced workload instances, changed downstream versions, new dependencies, removed dependencies, or endpoint changes. This filters the search space of the incident analysis to surface the small number of relationships that changed rather than scanning every dependency and infrastructure object associated with the service.
In this step, we move the investigation to the changed entities and their immediate neighbors. Signs of service degradation can be correlated with changes in entity relationships. This analysis allows us to highlight possible causal connections between change events to production incidents while filtering noise and red-herring incidents.
From the linked incidents, we can construct a chronological sequence of events that puts the topology states into an incident timeline. The topology explains what relationships existed, and the timeline explains when transitions occurred relative to the behavioral change.
The evidence we obtain from the reconstruction points out reasonable candidates for investigation i.e those events and entities which are most likely to have been the cause of the incident. However, this should be bounded by the understanding that temporal topology improves correlation, but it does not turn correlation into causality. In modern production systems with highly interdependent linking, many state changes may occur simultaneously which have no deterministic impact on service health. When two changes occur within seconds of each other, the evidence may support proximity without supporting a precise causal sequence. A prudent root cause analysis should distinguish direct observations, inferred relationships, temporal correlations, and causal hypotheses. These relationships should be investigated deeper to confirm dependency either by engineering context or experimental validation.
The core contribution of incident-time topology is not another visualization. Temporal reconstruction of relationships is a way to systematically utilize operational data to narrow down potential causes of production incidents more accurately and quickly.
The primary merit of this approach is reducing the number of things an engineer has to investigate to only relationships that changed around the incident. We also preserve ephemeral runtime context that would otherwise be ignored or lost. Post-incident analysis and handoffs become more robust when supplemented with historical relationship mappings. Good engineering analysis is still at the root of utilizing this tool to separate anomalies from truly impacting causal events.
The useful workflow is straightforward at a conceptual level: identify the incident time, reconstruct a known-good and incident topology, calculate the structural difference, correlate changed entities with behavioral telemetry, and use that evidence to form a bounded set of hypotheses.
A deployment can replace pods, a scheduler can move workloads, endpoints change, and service-to-service traffic may shift while an incident is still developing. The state of the service that an investigating engineer sees after pulling up dashboards may already differ from the state that produced the first symptoms. Any modern observability system can provide metrics, traces, and change logs for a root cause analysis. The analytical problem faced by the investigator is not whether there is sufficient historical context or visibility but instead
understanding what changes took place between the current degraded state and the last known healthy state of the system.
Essentially, it becomes an issue of reconstruction: whether we can reconstruct enough of the incident-time topology to understand what changed around the failure and then correlate those structural changes with behavioral evidence.
Why a Current-State Dependency Graph Is Not Enough
A conventional dependency graph to capture architecture relationships can be represented as G = (V, E), where vertices represent entities and edges represent relationships. For incident analysis, the additional dimension of time models the abstraction as G(t) to capture the relationships that were valid at a specific point in time. This distinction is important to tracing how the system changed over a sequence of events. This time-aware topology model treats those connections as temporal state.
If t1 represents the last known-good state and t2 represents the beginning of degradation, the difference between G(t1) and G(t2) identifies structural changes that occurred around the incident. While the changes are not necessarily root causes, they constitute a focused set of contributors that can be evaluated using telemetry and other evidence.
What Topology Information Matters for an Incident?
Incident analysis does not need to carry a comprehensive record of each relationship of the service in the production environment. Holding onto data that will likely not produce insights will inevitably increase operational overhead; to reduce noise and be economical with service costs, it is prudent to target critical information only. The useful topology is the subset that explains the operating context of the affected service, its logical dependencies, the workload instances serving it, where those workloads were running, as well as any infrastructure relationships that could materially influence behavior.
The right set of data to look for ultimately depends on the incident. A latency change may need just a service-level dependency graph to investigate while a placement-sensitive incident may require the service to be expanded into pods and nodes. We must be cognizant of this multi-resolution approach without turning it into a high-cardinality record of every runtime interaction for every investigation.
Reconstructing the Incident and Correlating Behavior
The most value we get from reconstruction is in comparing topology states and then evaluating the changed entities against behavioral telemetry. By correlating state changes to particular behavior, we can begin to build a causal analysis, almost a frame-by-frame playback of production incidents. The following framework provides a generic breakdown of the application of this approach.
Step 1: Establish the incident time
For investigation, the incident timestamp should be identified as the first observable point when service degradation began, which may not be when the underlying fault first occurred. This is not the alert timestamp, which is often the point after which a certain threshold has been breached for some time. The first indication of the behavioral change that led to the incident is a better marker, which allows us to correlate other factors that could have influenced a change in behavior.
In the representative example, Checkout latency begins increasing at 10:35 even though the alert does not fire until 10:38.
Tincident = 10:35
Step 2: Choose a known-good comparison point
To measure the deviation in the degraded system, we must establish a point of comparison with a state that is known to be healthy. The correct window before the incident before selecting a stable period is highly subjective, volatile systems will need to be baselined against a longer period while relatively steady ones can be traced closer to the degradation incident.
Tgood = 10:30
Step 3: Reconstruct both topology states
Taking each of the above timestamps, we establish the relationships for the service at each period. The reconstruction should be scoped to target pertinent information and expand into auxiliary dependencies only as required. i.e application dependencies, workload instances, placement and infrastructure should be pulled in on a case-by-case basis. This reconstruction depends on having enough historical relationship data available for those timestamps; missing records can limit what we can compare.
Ggood = G(10:30)
Gincident = G(10:35)
Step 4: Calculate the topology difference
The topology mappings provide a directed view of the service states that highlight changes in entities and relationships, which include placement transitions, replaced workload instances, changed downstream versions, new dependencies, removed dependencies, or endpoint changes. This filters the search space of the incident analysis to surface the small number of relationships that changed rather than scanning every dependency and infrastructure object associated with the service.
Step 5: Overlay behavior on the changed entities
In this step, we move the investigation to the changed entities and their immediate neighbors. Signs of service degradation can be correlated with changes in entity relationships. This analysis allows us to highlight possible causal connections between change events to production incidents while filtering noise and red-herring incidents.
Step 6: Reconstruct the sequence
From the linked incidents, we can construct a chronological sequence of events that puts the topology states into an incident timeline. The topology explains what relationships existed, and the timeline explains when transitions occurred relative to the behavioral change.
Step 7: Form a candidate hypothesis, not a verdict
The evidence we obtain from the reconstruction points out reasonable candidates for investigation i.e those events and entities which are most likely to have been the cause of the incident. However, this should be bounded by the understanding that temporal topology improves correlation, but it does not turn correlation into causality. In modern production systems with highly interdependent linking, many state changes may occur simultaneously which have no deterministic impact on service health. When two changes occur within seconds of each other, the evidence may support proximity without supporting a precise causal sequence. A prudent root cause analysis should distinguish direct observations, inferred relationships, temporal correlations, and causal hypotheses. These relationships should be investigated deeper to confirm dependency either by engineering context or experimental validation.
What Incident-Time Topology Changes About Investigation
The core contribution of incident-time topology is not another visualization. Temporal reconstruction of relationships is a way to systematically utilize operational data to narrow down potential causes of production incidents more accurately and quickly.
The primary merit of this approach is reducing the number of things an engineer has to investigate to only relationships that changed around the incident. We also preserve ephemeral runtime context that would otherwise be ignored or lost. Post-incident analysis and handoffs become more robust when supplemented with historical relationship mappings. Good engineering analysis is still at the root of utilizing this tool to separate anomalies from truly impacting causal events.
The useful workflow is straightforward at a conceptual level: identify the incident time, reconstruct a known-good and incident topology, calculate the structural difference, correlate changed entities with behavioral telemetry, and use that evidence to form a bounded set of hypotheses.
The progression in incident analysis