Assist In Root Cause Analysis | 🏗️ Build A Debugging Dashboard

Exam Guide: Developer – Associate
🏗️ Domain 4: Troubleshooting And Optimization
📘 Task 1: Assist In A Root Cause Analysis

When something breaks in production, you need to find out why. Fast. This task tests your ability to use CloudWatch Logs, Logs Insights, CloudWatch metrics, dashboards, and CloudTrail to debug application issues. Writing Logs Insights queries. Understanding Lambda-specific log fields. Using Embedded Metric Format for custom metrics. And recognizing common failure patterns across AWS service integrations.

📘 Concepts

CloudWatch Logs Architecture

Component What It Is Example Log Group A collection of log streams that share retention and access settings /aws/lambda/my-function Log Stream A sequence of log events from a single source 2024/01/15/[$LATEST]abc123 Log Event A single log entry with a timestamp and message {"level":"ERROR","message":"DynamoDB timeout"} Retention How long logs are kept (1 day to 10 years, or never expire) 30 days for production Subscription Filter Real-time stream of log events to another destination Stream to Kinesis, Lambda, or OpenSearch Metric filter Extract metric values from log text patterns Count ERROR occurrences

💡 Lambda automatically creates a log group named /aws/lambda/{function-name}. Each function instance creates a new log stream. Log retention defaults to “Never expire”. Always set a retention policy to control costs. You need logs:CreateLogGroup and logs:PutLogEvents permissions in the Lambda execution role.

CloudWatch Logs Insights Query Syntax

Logs Insights is a query language for searching and analysing log data. The exam tests specific query commands:

Command Purpose Example fields Select which fields to display fields @timestamp, @message filter Filter log events by condition filter @message like /ERROR/ stats Aggregate data stats count(*) by bin(5m) sort Order results sort @timestamp desc limit Cap the number of results limit 20 parse Extract fields from unstructured log text parse @message "user=* action=*" as user, action display Choose which fields appear in results display @timestamp, @message

Common Query Patterns

Find the most recent errors

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 25

Enter fullscreen mode Exit fullscreen mode

Count errors per 5-minute window

filter @message like /ERROR/
| stats count(*) as errorCount by bin(5m)
| sort errorCount desc

Enter fullscreen mode Exit fullscreen mode

Find slow Lambda invocations

filter @duration > 5000
| fields @requestId, @duration, @billedDuration, @memorySize, @maxMemoryUsed
| sort @duration desc
| limit 10

Enter fullscreen mode Exit fullscreen mode

Parse custom log fields

parse @message '"level":"*","requestId":"*","error":"*"' as level, reqId, error
| filter level = "ERROR"
| stats count(*) by error

Enter fullscreen mode Exit fullscreen mode

Find cold starts

filter @type = "REPORT" and @initDuration > 0
| fields @requestId, @initDuration, @duration, @memorySize
| sort @initDuration desc
| limit 20

Enter fullscreen mode Exit fullscreen mode

💡The parse command uses glob patterns (*) to extract fields from unstructured text. The stats command supports count(), sum(), avg(), min(), max(), pct() (percentiles). Use bin() to group by time intervals. The @ prefix denotes system-generated fields like @timestamp, @message, @logStream, etc.

Lambda-Specific Log Fields

Lambda automatically adds these fields to REPORT lines:

Field What It Contains Why It Matters @type Log line type: START, END, REPORT Filter for REPORT to get invocation summaries @requestId Unique invocation ID Correlate logs for a single invocation @duration Actual execution time (ms) Performance monitoring @billedDuration Rounded up to nearest ms (or 1ms minimum) Cost tracking @memorySize Configured memory (MB) Right-sizing analysis @maxMemoryUsed Peak memory used (MB) Detect memory pressure @initDuration Cold start initialization time (ms) Only present on cold starts @xrayTraceId X-Ray trace ID Link logs to traces

Memory right-sizing query

filter @type = "REPORT"
| stats avg(@maxMemoryUsed) as avgMem,
        max(@maxMemoryUsed) as peakMem,
        avg(@memorySize) as configuredMem,
        avg(@maxMemoryUsed) / avg(@memorySize) * 100 as utilizationPct

Enter fullscreen mode Exit fullscreen mode

💡 If @maxMemoryUsed is consistently close to @memorySize, the function may be running out of memory. If it’s consistently low (under 50%), you’re over-provisioned and wasting money. The @initDuration field only appears on cold starts. If you see it frequently, consider provisioned concurrency.

CloudWatch Embedded Metric Format (EMF) vs PutMetricData

There are two ways to publish custom metrics.

Aspect Embedded Metric Format (EMF) PutMetricData API How It Works Write structured JSON to stdout: CloudWatch extracts metrics automatically Call the CloudWatch API directly Latency Async (written with logs) Sync (API call blocks execution) Cost Log ingestion cost only (no per-metric API charge) $0.01 per 1,000 PutMetricData API calls Dimensions Up to 30 dimensions Up to 30 dimensions Resolution Standard (60s) or high (1s) Standard (60s) or high (1s) Batching Automatic (one log line = one metric data point) Manual (up to 1,000 values per API call) Lambda Overhead None. Just a print statement API call adds latency to each invocation Best For Lambda functions, high-throughput applications EC2 applications, infrequent metric publishing

EMF format

import json

def lambda_handler(event, context):
    # Your business logic here
    order_total = process_order(event)

    # Emit EMF metric. CloudWatch extracts this automatically
    print(json.dumps({
        "_aws": {
            "Timestamp": 1234567890000,
            "CloudWatchMetrics": [{
                "Namespace": "OrderService",
                "Dimensions": [["Environment", "OrderType"]],
                "Metrics": [
                    {"Name": "OrderTotal", "Unit": "None"},
                    {"Name": "ProcessingTime", "Unit": "Milliseconds"}
                ]
            }]
        },
        "Environment": "production",
        "OrderType": "standard",
        "OrderTotal": order_total,
        "ProcessingTime": 145
    }))

Enter fullscreen mode Exit fullscreen mode

PutMetricData approach

import boto3

cloudwatch = boto3.client('cloudwatch')

def lambda_handler(event, context):
    order_total = process_order(event)

    # Sync API call — adds latency
    cloudwatch.put_metric_data(
        Namespace='OrderService',
        MetricData=[{
            'MetricName': 'OrderTotal',
            'Value': order_total,
            'Unit': 'None',
            'Dimensions': [
                {'Name': 'Environment', 'Value': 'production'},
                {'Name': 'OrderType', 'Value': 'standard'}
            ]
        }]
    )

Enter fullscreen mode Exit fullscreen mode

💡> EMF is the preferred approach for Lambda because it adds zero latency. You’re just writing to stdout, which Lambda already sends to CloudWatch Logs. CloudWatch automatically extracts the metrics from the structured JSON. Use PutMetricData when you need to publish metrics from EC2 or when you need more control over metric timing.

Common Integration Failure Patterns

Scenarios to identify the root cause. Know these patterns:

Symptom Likely Cause How to Debug Fix 502 Bad Gateway (API GW) Lambda returned invalid response format Check Lambda logs for response structure Return statusCode, headers, body 504 Gateway Timeout (API GW) Lambda exceeded API Gateway’s 29-second timeout Check @duration in logs Reduce Lambda timeout, optimize code, or use async 429 Too Many Requests Lambda concurrency throttled or API GW rate limit Check Throttles metric Increase reserved concurrency or request limit increase Access Denied Missing IAM permissions Check CloudTrail for AccessDenied events Update IAM policy Task timed out (Lambda) Function exceeded its configured timeout Check @duration vs timeout setting Increase timeout or optimize code Out of memory Function exceeded configured memory Check @maxMemoryUsed vs @memorySize Increase memory allocation Connection timeout Lambda in VPC can’t reach internet or service Check VPC config, NAT Gateway, security groups Add NAT Gateway or VPC endpoint ResourceNotFoundException DynamoDB table or resource doesn’t exist in the region Check region configuration Verify resource exists in the correct region

502 Bad Gateway Deep Dive

The most common Lambda + API Gateway error. API Gateway expects this exact response format:

# CORRECT: API Gateway accepts this
return {
    'statusCode': 200,
    'headers': {'Content-Type': 'application/json'},
    'body': json.dumps({'message': 'success'})  # body must be a STRING
}

# WRONG: causes 502
return {
    'statusCode': 200,
    'body': {'message': 'success'}  # body is a dict, not a string!
}

# WRONG: causes 502
return "success"  # not the expected format at all

# WRONG: causes 502 (unhandled exception)
def handler(event, context):
    raise Exception("oops")  # no try/except = 502

Enter fullscreen mode Exit fullscreen mode

💡 A 502 from API Gateway almost always means the Lambda function returned a response that doesn’t match the expected format, or the function threw an unhandled exception. Check the Lambda logs first: the error will be there. The API Gateway execution logs (if enabled) will show “Malformed Lambda proxy response.”

CloudTrail for Debugging Permission Issues

CloudTrail Feature What It Does Use Case Event History Last 90 days of management events (free) Quick lookup of recent API calls Trail Continuous logging to S3 (all events) Long-term audit, compliance Data Events S3 object-level and Lambda invocation events Debug specific resource access Insights Detect unusual API activity Identify anomalous patterns

Finding AccessDenied events

1. Open CloudTrail console → Event history
2. Filter by Event name or Error code
3. Look for events with errorCode: AccessDenied or errorCode: UnauthorizedAccess
4. The event details show:

• userIdentity: who made the call (which role/user)
• eventName: what API was called (e.g., PutItem)
• requestParameters: what resource was targeted
• errorMessage: the specific permission that’s missing

💡 CloudTrail logs every AWS API call. When you get an AccessDenied error, CloudTrail tells you exactly which principal tried to call which API on which resource and which permission was missing. This is the fastest way to debug IAM issues. Event history is free and covers the last 90 days.

Deployment Failure Troubleshooting

Failure Type Where to Look Common Causes CloudFormation stack failure CloudFormation Events tab IAM permissions, resource limits, invalid template CodeBuild failure CodeBuild build logs Missing dependencies, test failures, buildspec syntax CodeDeploy failure CodeDeploy deployment logs Hook script errors, health check failures, timeout Lambda deployment failure CloudWatch Logs + CodeDeploy New version crashes, alarm triggers rollback SAM deploy failure CloudFormation Events + SAM CLI output Transform errors, packaging issues, capability missing

💡 CloudFormation failures show the specific resource and error in the Events tab. Look for CREATE_FAILED or UPDATE_FAILED status. The most common cause is insufficient IAM permissions because the CloudFormation role doesn’t have permission to create the resource. For SAM, make sure you include CAPABILITY_IAM and CAPABILITY_AUTO_EXPAND when deploying.

🏗️ Build A Debugging Dashboard

Build a Debugging Dashboard from scratch using the AWS Console:

• A Lambda function that emits structured logs and EMF custom metrics
• CloudWatch Logs Insights queries that find errors, slow invocations, and cold starts
• A CloudWatch dashboard displaying key application metrics
• Simulated failures (timeout, permission denied, 502) with debugging walkthroughs
• CloudTrail queries to find access denied events

Prerequisites

• An AWS Account
• Python

Part I

Create a Lambda Function with Structured Logging and EMF

Step 01: Create the Function

Open the Lambda console → Create function

• Function name: DebugDemoFunction
• Runtime: Python 3.13

Click Create function

Step 02: Add Structured Logging with EMF

import json
import time
import os
import random

def emit_emf_metric(metric_name, value, unit, dimensions):
    """Emit a CloudWatch metric using Embedded Metric Format."""
    print(json.dumps({
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": "DebugDemo",
                "Dimensions": [list(dimensions.keys())],
                "Metrics": [{"Name": metric_name, "Unit": unit}]
            }]
        },
        **dimensions,
        metric_name: value
    }))

def lambda_handler(event, context):
    start_time = time.time()
    action = event.get('action', 'process')
    request_id = context.aws_request_id

    # Structured log entry
    log_entry = {
        "level": "INFO",
        "requestId": request_id,
        "action": action,
        "timestamp": time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
        "functionName": context.function_name,
        "memoryLimit": context.memory_limit_in_mb
    }

    try:
        if action == 'slow':
            # Simulate a slow operation
            delay = random.uniform(2, 5)
            log_entry["level"] = "WARN"
            log_entry["message"] = f"Slow operation detected: {delay:.1f}s"
            time.sleep(delay)

        elif action == 'error':
            # Simulate an error
            raise ValueError("Invalid order data: missing required field 'customerId'")

        elif action == 'oom':
            # Simulate high memory usage
            log_entry["message"] = "Processing large dataset"
            data = "x" * (50 * 1024 * 1024)  # 50MB string

        else:
            # Normal processing
            log_entry["message"] = "Order processed successfully"
            time.sleep(random.uniform(0.1, 0.5))

        print(json.dumps(log_entry))

        # Emit EMF metrics
        processing_time = (time.time() - start_time) * 1000
        emit_emf_metric("ProcessingTime", processing_time, "Milliseconds",
                        {"Environment": "demo", "Action": action})
        emit_emf_metric("SuccessCount", 1, "Count",
                        {"Environment": "demo", "Action": action})

        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'Processed successfully',
                'action': action,
                'requestId': request_id,
                'processingTime': f"{processing_time:.0f}ms"
            })
        }

    except Exception as e:
        processing_time = (time.time() - start_time) * 1000
        log_entry["level"] = "ERROR"
        log_entry["error"] = str(e)
        log_entry["errorType"] = type(e).__name__
        print(json.dumps(log_entry))

        emit_emf_metric("ErrorCount", 1, "Count",
                        {"Environment": "demo", "Action": action})

        return {
            'statusCode': 500,
            'body': json.dumps({
                'error': str(e),
                'requestId': request_id
            })
        }

Enter fullscreen mode Exit fullscreen mode

Click Deploy

Step 03: Generate Test Data

Go to the Test tab

Create and run these test events multiple times:

Normal processing

{"action": "process"}

Enter fullscreen mode Exit fullscreen mode

Slow operation

{"action": "slow"}

Enter fullscreen mode Exit fullscreen mode

Error case

{"action": "error"}

Enter fullscreen mode Exit fullscreen mode

⚠️ Run each test event 5-10 times to generate enough log data

Part II

Write CloudWatch Logs Insights Queries

Step 01: Open Logs Insights

Open the CloudWatch console → ** ▼ Logs** → Log Management

Select the log group: /aws/lambda/DebugDemoFunction

⚠️ Set the time range to the last 1 hour

→ View in Logs Insights

Query 01: Find All Errors

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 25

Enter fullscreen mode Exit fullscreen mode

Click Run query. You should see the structured error logs from the error test events.

Query 02: Analyze Invocation Performance

filter @type = "REPORT"
| stats avg(@duration) as avgDuration,
        max(@duration) as maxDuration,
        min(@duration) as minDuration,
        count(*) as invocations

Enter fullscreen mode Exit fullscreen mode

Query 03: Find Cold Starts

filter @type = "REPORT" and @initDuration > 0
| fields @requestId, @initDuration, @duration, @maxMemoryUsed
| sort @initDuration desc

Enter fullscreen mode Exit fullscreen mode

Query 04: Parse Structured Logs for Error Breakdown

parse @message '"level":"*","requestId":"*"' as level, reqId
| filter level = "ERROR"
| stats count(*) as errorCount by level

Enter fullscreen mode Exit fullscreen mode

Query 05: Memory Utilization Analysis

filter @type = "REPORT"
| stats avg(@maxMemoryUsed) as avgMemUsed,
        max(@maxMemoryUsed) as peakMemUsed,
        avg(@memorySize) as configuredMem,
        avg(@maxMemoryUsed) / avg(@memorySize) * 100 as utilizationPct

Enter fullscreen mode Exit fullscreen mode

Query 06: Slow Invocations Over Time

filter @type = "REPORT"
| stats avg(@duration) as avgDuration,
        pct(@duration, 95) as p95Duration,
        pct(@duration, 99) as p99Duration
        by bin(5m)
| sort bin(5m) asc

Enter fullscreen mode Exit fullscreen mode

💡 Save useful queries by clicking **Save in Logs Insights. You can also add query results directly to a CloudWatch dashboard. The pct() function calculates percentiles: pct(@duration, 95) gives you the 95th percentile duration, which is more useful than averages for identifying tail latency.**

Part III

Create a CloudWatch Dashboard

Step 01: Build the Dashboard

Open the CloudWatch console → Dashboards → Create dashboard

Dashboard name: DebugDemo-Dashboard

Click Create dashboard

Step 02: Add an Invocation Count Widget

Click Add widget → Number

Select Lambda → By Function Name → DebugDemoFunction → Invocations

Period: 5 minutes

Click Create widget

Step 03: Add an Error Rate Widget

Click Add widget → Line

Add two metrics:

• Lambda → By Function Name → DebugDemoFunction → Errors
• Lambda → By Function Name → DebugDemoFunction → Invocations

Click Create widget

Step 04: Add a Duration Widget

Click Add widget → Line

Select Lambda → By Function Name → DebugDemoFunction → Duration

Add statistics: Average, p99, Maximum

Click Create widget

Step 05: Add a Custom EMF Metric Widget

Click Add widget → Number

Select DebugDemo (custom namespace) → Environment, Action → ProcessingTime

Click Create widget

Step 06: Add a Logs Insights Query Widget

Click Add widget → Logs table

Select log group: /aws/lambda/DebugDemoFunction

Enter query:

filter @message like /ERROR/
| fields @timestamp, @message
| sort @timestamp desc
| limit 10

Enter fullscreen mode Exit fullscreen mode

Click Create

Step 07:* Click **Save dashboard

Part IV

Simulate and Debug Common Failures

Step 01: Simulate a 502 Bad Gateway

Create a new Lambda function: BadResponseFunction and add this intentionally broken code

import json

def lambda_handler(event, context):
    # BUG: body should be a string, not a dict
    return {
        'statusCode': 200,
        'body': {'message': 'this will cause a 502'}  # Wrong!
    }

Enter fullscreen mode Exit fullscreen mode

Create an API Gateway REST API pointing to this function

💡 Test the API endpoint. You’ll get a 502

⚠️ Debug steps:

• Check Lambda logs → the function executed successfully (no error)
• Check API Gateway execution logs → “Malformed Lambda proxy response”
• Fix: Change 'body': {'message': '...'} to 'body': json.dumps({'message': '...'})

Step 02: Simulate a Timeout

On DebugDemoFunction, go to Configuration → General configuration → Edit

Set Timeout to 3 seconds

Run the slow test event (which sleeps 2-5 seconds)

⚠️ Debug steps:

• Check logs → Task timed out after 3.00 seconds
• Check the REPORT line → @duration close to 3000ms
• Query in Logs Insights:

   filter @message like /Task timed out/
   | fields @requestId, @duration, @memorySize

Enter fullscreen mode Exit fullscreen mode

Fix: Increase timeout or optimize the slow operation

Step 03: Simulate Permission Denied

On DebugDemoFunction, update the code to call DynamoDB:

import boto3
import json

dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):
    table = dynamodb.Table('NonExistentTable')
    try:
        table.put_item(Item={'PK': 'test', 'SK': 'test'})
    except Exception as e:
        print(json.dumps({
            "level": "ERROR",
            "error": str(e),
            "errorType": type(e).__name__
        }))
        return {
            'statusCode': 500,
            'body': json.dumps({'error': str(e)})
        }

Enter fullscreen mode Exit fullscreen mode

💡Run the test. You’ll get an AccessDeniedException

Debug steps:

• Check Lambda logs → AccessDeniedException
• Open CloudTrail → Event history
• Filter by Event name: PutItem
• Find the event with errorCode: AccessDenied
• The event shows which role tried to access which resource

Part V

Use CloudTrail to Find Access Denied Events

Step 01: Search Event History

Open the CloudTrail console → Event history

Change the filter to Error code

Type AccessDenied and press Enter

You should see the PutItem event from the previous step

Step 02: Examine the Event Details

Click on the event to expand it

Key fields to examine

• userIdentity.arn: the Lambda execution role that made the call
• eventName: PutItem
• requestParameters: the table name and item
• errorCode: AccessDeniedException
• errorMessage: the specific permission needed

💡To fix: add dynamodb:PutItem permission to the Lambda execution role for the specific table ARN

Step 03: Create a Logs Insights Query for CloudTrail

If you have a CloudTrail trail sending logs to CloudWatch:

fields eventTime, eventName, errorCode, errorMessage,
       userIdentity.arn as principal,
       requestParameters.tableName as table
| filter errorCode = "AccessDenied"
| sort eventTime desc
| limit 20

Enter fullscreen mode Exit fullscreen mode

💡 CloudTrail Event history is free and covers the last 90 days of management events. For longer retention or data events (S3 object access, Lambda invocations), you need a trail. When debugging permission issues, CloudTrail is your first stop, it tells you exactly what permission is missing, who tried to use it, and on which resource.

🏗️ What You Built | 📘 Exam Concepts Recap

What You Built Exam Concept Emitted EMF metrics from structured logs Zero-latency custom metrics via Embedded Metric Format Wrote CloudWatch Logs Insights queries fields, filter, stats, parse, sort, pct() syntax Queried @initDuration for cold starts Reading Lambda REPORT fields for performance Built a CloudWatch dashboard with mixed widgets Single pane of glass for application health Reproduced and fixed a 502 Bad Gateway Malformed Lambda proxy response (body must be string) Reproduced a timeout and read the REPORT line Diagnosing Task timed out from logs Found AccessDenied events in CloudTrail Debugging permission issues: principal, API, resource

⚠️ Clean Up Protocol

1. Lambda → Delete DebugDemoFunction and BadResponseFunction
2. API Gateway → Delete any test APIs created
3. CloudWatch → Delete the DebugDemo-Dashboard
4. CloudWatch → Delete log groups for the Lambda functions
5. IAM → Delete Lambda execution roles

Key Takeaways for the Exam

1. CloudWatch Logs Insights uses fields, filter, stats, parse, sort, and limit commands. Know the syntax for each.
2. Lambda REPORT lines contain @duration, @billedDuration, @memorySize, @maxMemoryUsed, and @initDuration (cold starts only). Use these for performance analysis.
3. EMF is preferred over PutMetricData for Lambda because it adds zero latency. Metrics are extracted from structured JSON written to stdout. Use PutMetricData for EC2 or infrequent publishing.
4. 502 Bad Gateway from API Gateway almost always means the Lambda response format is wrong: body must be a string, not a dict.
5. 504 Gateway Timeout means Lambda exceeded API Gateway’s 29-second limit. The Lambda timeout must be less than 29 seconds for synchronous API Gateway integrations.
6. CloudTrail logs every AWS API call. Use Event history to find AccessDenied errors. It shows the principal, API, resource, and missing permission.
7. Metric filters extract CloudWatch metrics from log text patterns. Subscription filters stream log events to Kinesis, Lambda, or OpenSearch in real time.
8. CloudWatch dashboards can include metrics widgets, Logs Insights query results, and alarms giving you a single pane of glass for application health.
9. The parse command in Logs Insights extracts fields from unstructured text using glob patterns (*). The stats command supports count(), avg(), sum(), min(), max(), and pct() for percentiles.
10. When debugging deployment failures, check CloudFormation Events (stack failures), CodeBuild logs (build failures), and CodeDeploy logs (deployment failures). Each service has its own log location.

Additional Resources

• CloudWatch Logs Insights language query syntax
• Specification: Embedded metric format
• Monitoring, debugging, and troubleshooting Lambda functions
• Working with CloudTrail event history
• Creating a customized CloudWatch dashboard
• Troubleshooting issues in Lambda

🏗️