Cloud Security Troubleshooting Playbook

Ask yourself: can another engineer read your first five minutes of notes and understand what happened, what remains unknown, and what action you have avoided? If not, your investigation needs structure before it needs more tools.

Build the 5W1H Triage Map

The 5W1H Triage Map turns an alert into a bounded investigation. Fill it from observed evidence, not assumptions. Mark each statement as confirmed, probable, or unknown. That distinction matters because severity decisions often fail when a probable explanation gets recorded as fact.

1. Who identifies the actor and the credentials involved. Check the human user, service principal, managed identity, role session, access key, or workload account. In Azure, review the Microsoft Entra ID sign-in and audit records. In AWS, review AWS CloudTrail events and the principal identity. Confirm whether the identity normally accesses the account, subscription, region, or service.

2. What describes the action and its result. Separate attempted actions from successful actions. A denied request to read a storage bucket does not prove data exposure. A successful policy change, secret read, or role assumption requires a different response. Capture the API operation, resource name, response code, and any object or record identifiers.

3. When defines the investigation window. Use Coordinated Universal Time (UTC) in the incident record so teams in different locations compare events correctly. Start at least 30 minutes before the first alert and continue until the latest related activity. Extend the window when a credential may have existed for longer or when logs show earlier reconnaissance.

4. Where maps the activity to cloud account, subscription, region, virtual network, workload, and source location. A sign-in from a new country may matter less than an unexpected access path from a familiar corporate network. Conversely, an attacker using a valid virtual private network (VPN) exit point can make location alone unreliable. Treat location as one signal, not proof.

5. Why records the detection reason and the business context. An impossible-travel alert may result from a mobile network, proxy, or token reuse. A new administrator role may come from an approved deployment or from privilege escalation. Compare the event with change tickets, deployment records, and on-call activity before assigning intent.

6. How explains the access path and the likely technique. Determine whether the actor used a password, multifactor authentication (MFA) session, access key, token, role assumption, automation pipeline, or exposed endpoint. Then check what permissions that path allowed. The same suspicious source produces different risk when it reaches a read-only test account versus a production administrator role.

Classify the alert after you populate the map. Use three practical states: benign or expected, suspicious but unconfirmed, and confirmed security incident. Mark an alert benign only when evidence explains the event and shows no unauthorized result. Mark it suspicious when the actor or behavior lacks a trusted explanation, but successful impact remains unproven. Mark it confirmed when logs show unauthorized access, a successful destructive or privileged action, data exposure, persistence, or a control failure with credible evidence of exploitation.

Then scope impact across four dimensions: identities, resources, data, and time. For example, “one service principal accessed one storage account” is incomplete. Record whether the principal could reach other subscriptions, whether the storage account held sensitive data, whether access succeeded, and how long the credential remained usable. Scope determines containment priority and prevents teams from rotating every credential when one identity requires immediate action.

Keep the first response read-only whenever possible. Export logs, preserve alert details, and query activity before changing configuration. If you need containment, record the exact action, operator, time, expected effect, and rollback method. A disabled account may stop an attacker, but it may also stop an application and remove the session behavior you still need to observe.

Apply the Map to a Production Alert

A production alert reports a successful role assignment in an AWS account at 14:07 UTC. The event names a deployment role, originates from an unfamiliar address, and grants administrator access to a new principal. The following workflow produces an initial scope without assuming that the role assignment caused data access.

1. 14:10 UTC - preserve the alert. Export the alert JSON, copy the event identifier, and record the account ID, region, principal, source address, and detection rule. Expected outcome: the original evidence remains available even if the alert later closes or the dashboard changes.

2. 14:13 UTC - query CloudTrail. Search 30 minutes before and 30 minutes after 14:07 UTC for the event identifier, principal, source address, and role name. Look for CreatePolicy, AttachRolePolicy, AssumeRole, storage reads, key-management actions, and logging changes. Expected outcome: you determine whether the role assignment was the only successful action or part of a larger sequence.

3. 14:18 UTC - validate the actor. Compare the source address and user agent with the deployment system’s known egress addresses. Check the deployment record for a change approved at 14:07 UTC. Expected outcome: you either connect the event to an approved deployment or document that no trusted explanation exists.

4. 14:23 UTC - measure permissions. Inspect the new principal’s policies and the deployment role’s ability to create or modify identities, storage access, encryption keys, logging, and network controls. Do not test permissions by making changes. Expected outcome: you know the maximum credible blast radius without creating additional events.

5. 14:28 UTC - check use of the new privilege. Search CloudTrail for the new principal and any sessions created after the role assignment. Check data access logs for storage reads and key-management logs for decrypt operations. Expected outcome: you distinguish privilege creation from actual resource or data access.

6. 14:35 UTC - assign an initial severity. If the event matches an approved deployment and no unexpected action occurred, classify it as benign or expected and attach the change record. If the source is unknown but no successful follow-on action appears, classify it as suspicious and escalate for containment planning. If the principal accessed sensitive data, disabled logging, or created persistence, classify it as a confirmed incident and activate the production response plan.

7. 14:40 UTC - choose the least disruptive containment. For confirmed unauthorized use, revoke the affected session or disable the specific principal, then remove the unauthorized policy or role assignment. Preserve the original policy document and event records first. Expected outcome: the access path closes while investigators retain evidence and can explain each change.

8. After containment - verify. Run the same CloudTrail searches for the next 30 minutes, confirm that new sessions fail, and check that logging still records denied attempts. Expected outcome: the response stops the identified path and does not hide continuing activity through another credential.

Record the result in a compact incident table:

| Field | Example | |---|---| | Classification | Suspicious, unconfirmed | | First observed | 14:07 UTC | | Identity | Deployment role and new principal | | Confirmed action | Administrator policy attached | | Confirmed impact | No data access found in current window | | Unknowns | Earlier credential use; other regions | | Next action | Expand search to 24 hours and validate deployment source | | Owner | Cloud security on call |

Quick checklist

• Preserve the raw alert and event identifiers. - Set the time window in UTC. - Separate attempted actions from successful actions. - Check identity, resource, data, and time scope. - Compare activity with approved changes and automation. - Use read-only queries before containment. - Record every containment change and rollback path. - Recheck logs after containment to confirm the access path stopped.

The key result is not a perfect explanation at 14:40 UTC. It is a reliable boundary around what you know, what you suspect, and what you must check next.

Avoid Triage Errors That Increase Risk

Treating the alert label as the incident

Detection rules compress many signals into a short title. “Impossible travel” may reflect token reuse, a proxy, or a legitimate network change. “Public storage exposure” may identify a policy state without proving that anyone accessed an object.

Do this: inspect the underlying event, result status, identity, resource, and time window. Not this: declare compromise from the alert title alone.

If the event shows a denied request, record the denial and search for successful requests by the same identity or source. Classification depends on observed results, not the severity wording chosen by a product.

Changing the environment before preserving evidence

Disabling an account, deleting a role, or rotating a key can stop an attacker, but it can also remove the session context that links events together. Emergency containment may take priority when active damage continues, yet you still need a minimum evidence set.

Do this: export alert details, preserve relevant audit logs, and note the current policy before changing it. Not this: delete the suspicious principal and investigate afterward.

If active access threatens sensitive data, record the decision, capture the best available evidence, and contain the narrowest confirmed path. Ask another responder to preserve logs while you execute the emergency action.

Assuming one identity means one scope

An attacker may use one compromised identity to assume roles, access several regions, or reach multiple subscriptions. Conversely, a broad alert may involve only one failed request. Scope the reachable resources and the resources actually used separately.

Do this: search for the identity, session name, source address, and related role assumptions across every account or subscription in the investigation window. Not this: investigate only the resource named in the first alert.

Pay special attention to control-plane actions such as policy changes, role assignments, logging changes, key access, and creation of new credentials. These actions can expand impact even when data access has not yet appeared in the logs.

A sound triage record gives the next responder a map rather than a mystery: who acted, what succeeded, when it happened, where it reached, why the alert fired, and how the access occurred. That map makes containment safer and turns the first response into evidence for the deeper identity, configuration, network, storage, logging, and data protection investigations that follow.

End of chapter one. 39 more chapters and about 73,044 words in the full book.