비전 모델로 14,512개의 문화유산 사진을 감사했습니다. ‘신뢰할 수 있는’ 출처가 가장 더러웠다.

Kahve Tabela is an open atlas of Türkiye’s cultural heritage: 32,000+ registered places — castles, ancient cities, mosques, museums — plus a coastal atlas and a regional-food atlas. Every record carries its sources, the corpus ships with a DOI on Zenodo, and the whole thing is free.

For a reference site, trust IS the product. So photos don’t just get scraped and shipped: every one of the 14,512 images on the atlas has been audited by a vision model before it’s allowed to represent a heritage site. One reader once reported a single mismatched photo — and instead of patching that one page, we built a pipeline that checks all of them, forever. This post is how that pipeline works, and two things it taught us that apply to any dataset or RAG corpus.

The setup: one Mac Studio, zero API bill

Qwen3-VL 30B running locally on MLX. Every image downscaled to 512px first — 10x throughput, and on a 512-photo validation set the verdicts didn’t measurably change. At that size, auditing 14,512 photos is an overnight job, not a budget line.

Lesson 1: error rate is a property of the SOURCE

Open-data pipelines pull images from many upstreams. Slice the audit by provenance and the upstreams are not remotely equal:

upstream source photos flagged by first pass Wikipedia article images (scraped from the article body) 300 64.3% Wikimedia (title/filename matched) 1,140 55.6% Wikimedia Commons geosearch 1,549 17.2% National heritage inventory (Kültür Envanteri) 6,210 9.5% Google Places 3,070 7.2% Mapillary 126 6.3%

The intuition “it’s on the Wikipedia article about the place, so it depicts the place” fails at scale: article bodies carry maps, portraits, neighboring buildings, the view from the site. If your pipeline scrapes article bodies, you inherit all of it. Ours doesn’t anymore — more on that below.

Lesson 2: a single model opinion is an accusation, not a verdict

Here’s the part most audit writeups skip: the auditor itself has to be audited. Before deleting anything we re-tried every flagged photo with a harder, humbler question. Pass 2 hid the place name entirely — the model saw only the image and the category: “does this show a castle or castle ruins? yes/no.”

Pass 2 upheld only 37% of pass 1’s flags. The transcripts show why: given a name, a model reasons about the NAME (“this record is in Denizli but the photo looks like Elazığ”) and happily hallucinates geography. Identity questions invite hallucination; category questions are name-proof.

With both passes required to agree independently:

upstream source confirmed mismatch Wikipedia article images 29.3% Wikimedia Commons geosearch 1.5% Wikimedia (title-matched) 1.2% National heritage inventory 1.2% Mapillary 0.8% Google Places 0.1%

Two-pass agreement is what saved ~600 perfectly good photographs a naive single-pass audit would have deleted — and it’s what makes the final numbers trustworthy enough to act on.

What the atlas does with this

• The 264 two-pass-confirmed mismatches (1.8% of the corpus) were removed the same week.
• Every image carries a provenance field, so the whole corpus can be re-audited as models improve.
• Ingestion is now trust-ranked: Google Places and Mapillary photos auto-attach; anything scraped from a Wikipedia article body goes to a human review queue first.
• The audit harness stays in the toolchain — new photos go through the same gate before they ever reach a page.

If you’re building a dataset (or a RAG pipeline)

1. Provenance is a feature. Store where every asset came from; error rate is a property of the source, and you can’t fix what you can’t slice.
2. Never audit with the name in the prompt. Ask what the model SEES, in a closed vocabulary, and compare against the record yourself.
3. One model opinion is an accusation, not a verdict. Require two independent passes with different framings to agree before you delete.
4. “It’s from the article about X” is not evidence it depicts X.

The atlas and its data are open: browse it at kahvetabela.com, grab the corpus on Zenodo (DOI, CC-layered licensing per source) or Kaggle, or point an agent at the read-only MCP server.

Methods note: pass 1 ran name+province+photo over all 14,512 records; pass 2 re-ran flagged photos name-blind with category-only questions on a 1,560-photo slice of the 2,118 flags. Confirmed-mismatch = flagged by pass 1 AND rejected name-blind by pass 2 AND surviving a human spot-check of edge buckets. Model: mlx-community Qwen3-VL-30B (8-bit), temperature 0.