In this video, we perform a deep dive into evaluating Retrieval-Augmented Generation (RAG) performance using Amazon Bedrock Knowledge Bases and the LLM-as-a-Judge framework.
Top comments (2)
Subscribe
Collapse Expand
Building AI-powered tools for developers. Creator of terminalskills.io — curated terminal skills and CLI tools for modern devs.
-
Location
Dallas-Fort Worth, Texas
-
Work
Founder at AIEmployees & Terminal Skills
-
Joined
Mar 7, 2026
• Aug 8
LLM-as-judge can be useful, but I would keep it as one layer, not the final truth. For RAG, the judge should inspect citation support, answer completeness, refusal behavior, and whether the retrieved context actually contains the claim.
Collapse Expand
AI engineer. I automate document work: translation, extraction, review, without wrecking your formatting.
-
Location
Seoul, Korea
-
Joined
Dec 26, 2025
• Aug 10
Alex’s citation support check is the one I’d add first. We ran into a case where our judge was scoring faithfulness at around 85% on a Bedrock KB, and when we spot-checked that week’s batch by hand the actual number was closer to 65%. The judge was systematically giving credit for answers that restated the question instead of citing the passage. We ended up calibrating against about 50 hand-labeled pairs from the same domain before we trusted the automated scores again.
Code of Conduct • Report abuse
For further actions, you may consider blocking this person and/or reporting abuse
답글 남기기