Interactive AI Eval Dashboards with Data Studio

Welcome to the final entry of our series about designing, analyzing and visualizing AI evals! As with the previous entries, you are more than welcome to simply read over my exploration or, for those of you looking for extra credit, feel free to play along at home. If this isn’t you, please skip to “Overview of New Pre-Configured Dashboard Components”

Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing. Furthermore, the math is mine but the formula formatting is AI’s.

Implementation prerequisites

If you intend to (re)implement the procedures in this walkthrough, all you need is the CSV export (data_mega_export.csv) generated during Part 3 (from evaluations performed in Part 1) and a Google Workspace account to access Data Studio. Part 2 will give you a deeper understanding of evals, trace diagnostics and back-of-the-napkin analysis but is optional if you just want to build dashboards.

Back to the story at hand

In this final part of our series, we connect our evaluation dataset to Data Studio to build codeless, interactive dashboards.

While inspect view provides developer-focused TUI trace diagnostics and Google Sheets gives us tabular sparklines, non-technical stakeholders (product managers, business leads, executive sponsors) need interactive dashboards where they can slice performance across our foundational 3D Evaluation Matrix on the fly without writing code. Beyond macro dimensions like model family and skill group, our dashboard empowers users to dynamically partition data over key operational booleans:

• completed (Conditional Accuracy): Isolates Model Capability (accuracy on completed runs) from infrastructure noise (quota drops, timeouts, container crashes). Toggling between TRUE and ALL exposes Survivorship Bias in task execution.
• is_baseline_or_has_activated_skill (Conditional Uptake Slicer): Separates baseline control cohorts from active skill interventions where the model invoked activate_skill (i.e. uptook the skill into context).
  • Crucially, “activated” in Inspect AI telemetry specifically tracks skill uptake (executing the activate_skill tool call). Filtering for FALSE isolates dormant runs where a skill was available but never uptaken.

1-click Data Studio template setup

We provide a pre-configured Master Dashboard Template containing all pre-styled components used here.

Clone the master template

1. Open the Inspect AI Dashboard Template.
2. Click Make a copy

30-second setup and customization

Once cloned:

1. In the new report, press Add data
2. Select Google Sheets
3. Select the Spreadsheet and the Worksheet (likely the sole Worksheet) you created in the third entry and press Add
4. Select all the components and set their data source to that sheet
5. Finally, select the bubble chart and open the Style tab, find the Bubble color dropdown and set it to ‘model’

That’s it! Your interactive dashboard is now live, fully bound to your evaluation data.

Before diving into the new charts, check out the bottom left corner of the report. There you will find a Pivot Table Heatmap. Like the heatmap I made in the second entry in this series, this heatmap is a 2D visual matrix where color intensity represents numerical magnitude—here, quadratic curved accuracy scores. Since I already discussed how to use these for analysis in the earlier blog entry, I will not be explaining it here. If, on the other hand, you are interested in implementation, feel free to inspect and adjust the settings on the Pivot Table Heatmap in your copied report.

Note: This screenshot was done on a later dataset than the rest of the report and the specific metrics should not be taken into account when comparing to the rest of the screenshots here.

Overview of new pre-configured dashboard components

Data Studio translates multi-dimensional evaluation CSV data into intuitive visual dimensions (spatial X/Y coordinates, bubble area scaling, color coding, and hover popovers). Here is how each pre-built dashboard element visually measures evaluation performance:

Chart A: Performance and cost (scatter/bubble chart)

This chart maps three core evaluation dimensions onto a single 2D visual canvas:

• Horizontal Position (X-Axis ➔ Latency): Measures average task duration in seconds. Bubbles placed further to the right represent slower runs.
• Vertical Position (Y-Axis ➔ Score / Accuracy): Measures average task accuracy from 0.0 to 1.0. Bubbles placed higher up represent better accuracy.
• Bubble Size / Area (Visual Cost & Unit Economics): Circle size scales with average task cost for a quick visual gut-check.
• Hover Tooltips & Sub-Cent Formatting: Hovering displays latency, accuracy, model, skill group, and cost formatted as Currency > USD (#,##0.0000) so sub-cent costs (e.g., $0.0044) don’t round to $0.00.

Chart B: Accuracy and error bars (±1 SEM whiskers)

The second pre-configured chart visualizes accuracy scores alongside standard error of the mean (SEM) whiskers.

• Visual Whisker Bounds: The bar height shows average accuracy (score), while floating dots above and below show upper (score_lower) and lower (score_upper) standard error bounds.
• Design Note (Multi-Metric vs. Breakdown Dimension Constraint): Data Studio won’t let you use a Breakdown Dimension if you’re plotting multiple Y-axis metrics on the same chart. Because our whiskers need three Y-metrics (score, score_lower, score_upper), we put model on the X-axis and filter by skill_group using top-level drop-downs instead.

Statistical Rigor Note: Standard error tracks pass/fail score variation across runs. Our demo ( N=4N=4 ) covers a ~68% confidence range ( ±1 SEM\pm 1\text{ SEM} ), while production benchmarks require N≥30N \ge 30 for standard 95% confidence intervals.

The 4 dynamic cohort slicers (drop-down controls)

Located across the top control bar, these four drop-down lists act as interactive dynamic filters, immediately re-rendering both canvas charts in real time:

1. model Slicer: Filters results by model family (gemini-3.5-flash-lite vs gemini-3.6-flash).
2. skill_group Slicer: Filters by skill setup (gemini-api, gcloud, baseline control).
3. is_baseline_or_has_activated_skill Slicer (Conditional Uptake): Separates Baseline controls and Active Skill runs (TRUE via activate_skill) from Dormant runs (FALSE, offered but uncalled). Activation confirms tool uptake, not instruction compliance.
4. completed Slicer (Conditional Accuracy / Infrastructure Noise): Filters clean runs (TRUE) from system failures (FALSE for quota drops, timeouts, container crashes).
   • Relevance Note: This is vital for production sweeps hit by rate limits or timeouts—selecting TRUE measures Model Capability (Conditional Accuracy), while FALSE or (All) shows the full audit trail.

View and interact

1. Click the blue View button in the top right corner to see the dashboard as a user would.
2. Use the model, skill_group, completed, and is_baseline_or_has_activated_skill drop-downs at the top.
3. Notice how both the Bubble Chart and the Bar Chart update instantly in unison to show only the models or skills you selected!

Slicer-based analyses

While the model slicer would be useful in a situation with more models, since we only have two and you can visualize them using the existing coloring. skill_group on the other hand is more interesting

On selecting gemini-api (Skill) and gemini-api (Baseline), there are a couple visually apparent trends. As discussed in previous sections, the gemini-api (Skill) run improved the evaluation score at the price of increased latency and cost.

Conditional uptake analysis: Separating skill discovery from capability

By filtering for runs where is_baseline_or_has_activated_skill is FALSE, we perform a Conditional uptake analysis: examining dormant runs where a skill was provided in context (Prior), but the agent chose not to call activate_skill (zero uptake).

Baseline or has activated skill = false (notice these both score relatively low)

Baseline or has activated skill = true

One thing to note immediately is that the total number of bubbles is GREATER than 8 (the count of Model x Skill combinations). This is because each of these bubbles are subsets of the Model x Skill x Sample x Epoch space in that this distinction is made at the samples/epochs level.

In Inspect AI telemetry, “activated” specifically measures uptake—whether the model executed the activate_skill tool call to load the skill into context. Interestingly, the samples of 3.5-flash-lite where the skills were activated have a much higher visual ceiling than the general case (Baseline or Has Activated Skill = Any) corresponding to the elimination of Baseline or Has Activated Skill = False.

Noting that only 3.5-flash-lite had samples where it didn’t activate a provided skill, it may be worth investigating if this is a more general trend.

Survivorship bias and joint operational uptime (completed analysis)

For the sake of further exploration, I reran the evals with web search enabled and the same time limit.

inspect eval skills-eval.py \
  --model google/gemini-3.5-flash-lite,google/gemini-3.6-flash \
  --time-limit 300 \
  --epochs 2 \
  --max-tasks 4 \
  -T web_access=true

Enter fullscreen mode Exit fullscreen mode

Running all of these evals serially and in parallel causes them to use too much web search quota and reach the unforgiving time-limit. As such, the most realistic run configuration would be to increase the time limit and use the built-in adaptive rate limits. The problem is this would last a lot longer and take more resources than I care to ask you to emulate.

So let’s look instead at the results of the web search enabled, 300 second time limit run:

completed: ALL

Since this graph contains every model x skill x sample x epoch, you can eyeball the average metric values of a number of bubbles by imagining the centroid of the polygon they define on the space.

Some of the most obvious conclusions in our local web-search enabled eval sweep: 3.5-flash-lite recorded lower latency than 3.6-flash, while their average scores were comparable. To reason about what this might mean, refer to the Gemini API Model Documentation for official model latency, throughput, and performance specifications. Instinctually, this makes me want to check if the timed out runs (counted as score 0) affected 3.6-flash more (since it didn’t complete as quickly in these samples). To test this, let’s see how scores look when we only count completed runs.

completed: TRUE

When filtering the web-search rerun sweep strictly for completed runs (completed = TRUE), the average score for the 3.6-flash sample cohort trended slightly higher, converging between 0.5 and 0.6 for partially scoring runs (which earlier averaged between 0.35 and 0.60 when completed = ALL).

Isolating completed = TRUE calculates Conditional Accuracy (Model Capability on completed runs). However, if longer or more complex tasks disproportionately trigger time-limits, filtering out incompletes introduces Survivorship Bias, truncating the difficult tail of your test set. Comparing completed = ALL against completed = TRUE bridges the gap between pure model intelligence and total operational system reliability for given limits.

Are these findings generalizable to every scenario?

While we’ve uncovered a promising trend, the next step is to validate it with statistical rigor. Think of our current demo ( N=4N=4 ) as an exploratory pilot; it’s great for spotting trends, but we need more data to confirm them. To verify these results aren’t coincidental, I encourage you to scale your runs to N≥30N \ge 30 . With this larger sample size, you can achieve statistical significance, a metric used to ensure that your findings aren’t just a random outlier. This is supported by the p-value—a standard calculation that gives you the probability that your results are a fluke. Aiming for a low p-value is the final step in moving from exploratory analysis to building the mathematically sound, launch-ready benchmark required for production.

Conclusion and next steps: Where to go from here

We’ve come full circle in this series—from running terminal-based evaluation sweeps with Inspect AI in Part 1, to analyzing multi-turn LLM reasoning traces in Part 2, modeling tabular metrics with pandas in Part 3, and finally assembling interactive, codeless BI dashboards here in Data Studio.

Now that you have the end-to-end telemetry pipeline running, the floor is yours to explore! Here are a few directions you can take your evaluation harness next:

• Custom Calculated Metrics: Experiment with custom formulas in Data Studio to track ROI metrics, cost per successful task, or normalized token efficiency ratios.
• Expanded Matrix Sweeps: Scale your evaluation matrix across new LLM provider endpoints, prompt techniques, or tool playbooks.
• Direct Database Connectivity: Replace the static CSV / Google Sheets ingestion layer with a direct connector to BigQuery, PostgreSQL, or Cloud Storage for automated, streaming evaluation updates.
• Automated Alerting & Thresholds: Set up automated notifications when a new model deployment drops below baseline accuracy or exceeds latency/cost budgets.

Thank you for following along with this series, and happy evaluating!