FaultTrellis

FIELD GUIDE / INCIDENT REPORTING

Make the incident
easy to follow.

A useful report gives the next responder a reliable account of what changed, when it changed, what the evidence showed, and what still needs attention.

Treat the report as a working record, not a verdict. Separate direct observations from hypotheses, use links for evidence held in other tools, and update the record when the team learns more. If a detail is uncertain, label it as an estimate or leave it open.

01 / SIGNALS AND IMPACT

Describe the signal in its service context.

Start with who or what was affected, then state the signal that brought the issue to attention. A number without scope or context is difficult to interpret after the response.

  • Name the affected service, endpoint, job, queue, region, or customer-facing workflow, and give the known scope of impact.
  • Identify the signal and its source: for example, application errors, latency, saturation, queue depth or age, failed jobs, or a deployment health check.
  • Include the observed value or condition, its time window, and the relevant baseline, threshold, or expected behavior when known.
  • Link to the authoritative dashboard, log query, alert, or change record. Avoid copying credentials, secrets, or unnecessary personal data into the report.

02 / TIMING AND EVIDENCE

Make the sequence and confidence clear.

Record the first known impact separately from when it was detected and when someone began responding. Use one timezone throughout, preferably UTC, and mark estimated times as estimates. If a note is added after the event, distinguish the event time from the time it was recorded.

Keep observations separate from explanations.

“Queue age increased after the release” is an observation with a sequence. “The release caused the backlog” is a hypothesis until evidence supports it. Label hypotheses and update them as the investigation proceeds.

Useful milestones include first observed impact, alert or report time, response start, each mitigation, first sign of improvement, recovery, and the end of any verification period. If the exact onset is unknown, record the last known healthy point and the first known affected point instead of inventing precision.

03 / RESPONSE CHANGES

Record what was proposed, approved, and done.

Each material response change should have its own timeline entry. State the intent and target, who proposed or authorized it, who or what carried it out, and the observed result. Keep these states distinct: a proposal is not approval, and approval is not execution.

  • Name the change and its scope, such as a rollback, configuration adjustment, traffic shift, queue pause, or dependency change.
  • Link a deployment, commit, change ticket, or other durable reference where one exists; avoid pasting large raw logs into the narrative.
  • Record whether it was proposed, approved, executed, rejected, rolled back, or not run. Include the reason when that distinction matters.
  • Note the result that was actually observed after the change, including any verification performed. If no change was made, say so plainly.

04 / UNRESOLVED ISSUES

Leave a clear handoff, including what is unknown.

A recovered service does not mean every risk or follow-up is closed. End the report with the current status and a short list of open work, temporary mitigations, and unanswered questions.

  • State any remaining user impact, degraded behavior, or monitoring concern.
  • Call out hypotheses that still need evidence and gaps in the available timeline.
  • Give each next step a responsible team or role, a target time set by your team, and a way to verify completion.
  • Link related follow-up work and state whether a temporary workaround needs to be removed or reviewed later.

05 / EXAMPLE TIMELINE

Keep every entry short and attributable.

These invented, relative entries illustrate a format only. Replace them with the incident’s observed facts and your team’s actual decisions.

ILLUSTRATIVE ONLY / NOT A REAL INCIDENT

  1. Observed — A service signal moved outside its usual range; source and affected scope linked.
  2. Investigating — The response team noted a possible relation to a recent change; cause not confirmed.
  3. Change recorded — A mitigation was approved and carried out; change reference and executor recorded.
  4. Verifying — The signal improved; the team continued checking related service behavior.
  5. Still open — A follow-up check remained, with a team owner and completion criterion recorded.

06 / REPORT OUTLINE

A compact starting point.

Adapt this outline to your incident process. Keep uncertain fields explicit, and remove anything your team does not use.

Summary:
Impact and scope:
Signals, sources, and comparison window:
First observed / detected / response started:
Timeline (timezone; event time and note time if different):
Changes (proposed / approved / executed / not run; by whom; result):
Current status and verification:
Unresolved risks or questions:
Next step — owner (team or role), target time, verification: