I Built a Security Scorer for 213 MCP Servers — Here's What Made the Difference

Every time I wanted to add an MCP server to Claude Desktop or Cursor, I ended up
with a dozen browser tabs open, trying to answer one question the README never
answers: is this actually safe to give tool-call access to my machine?

MCP (Model Context Protocol) directories exist, but most of them just mirror
GitHub — stars, a README excerpt, a copy-paste config. None of them score
risk. So I built RepoAI, and I want to share the exact
methodology, because I think it’s more interesting than the product pitch.

The problem with “just check the stars”

Stars measure popularity, not safety. A server with 5,000 stars can still:

• have no read-only mode (every tool call can write/delete)
• ship with no auth method at all
• be an unmaintained fork nobody’s patched in a year
• be archived — meaning whatever’s wrong with it stays wrong, forever

None of that shows up in a star count.

The 15 signals

computeSecurityScore() runs 15 checks against each server, sums signed point
values, and clamps the result to 0–100. Two categories:

Repo health (max +101):

Check Points Why Official maintainer +18 / 0 Vendor-published > random fork Editorially verified +5 / 0 A human actually reviewed it Actively maintained +12 / +6 / 0 Last commit ≤90d / ≤365d / older Established contributor base +10 / +5 / 0 Bus factor: ≥5 / ≥2 / 1 Community adoption +10 / +5 / 0 Stars: ≥1000 / ≥100 / fewer Has license +8 / 0 Unlicensed = unclear usage terms Reasonable dependency footprint +5 / +2 / 0 Smaller supply-chain surface Has homepage/docs +3 / 0 Substantial README +5 / 0 ≥500 chars Archived 0 / −40 No future fixes. Ever.

MCP-specific:

Check Points Why Read-only mode available +15 / 0 A safer way to run it should exist Supports OAuth +10 / 0 Stronger than static tokens Dangerous tools present 0 / −10 / −25 Share of tools flagged execute/delete/write Dangerous tools, no safe mode 0 / −20 The double penalty — risky and no escape hatch Weak or missing auth 0 / −15 Anyone with the config file can use it

Scores land in three bands: 80–100 safe, 60–79 medium, 0–59 high risk.

The part I almost got wrong

My first instinct was to let an LLM write the verdict and set the score.
That’s a mistake — an AI can be convinced by good marketing copy in a README
just as easily as a human can. So the pipeline splits it:

• The score comes only from verifiable facts: GitHub metadata, license, commit history, tool risk classification. No AI in that loop at all.
• The verdict (the human-readable “should you use this” paragraph) starts as an AI draft grounded strictly in those same facts — then a human reads it before it ever goes live. Nothing gets published on the strength of an AI’s opinion alone.

What it looks like end to end

For a server like qdrant/mcp-server-qdrant, the pipeline:

1. Scrapes the repo (README, package.json, releases, commit history)
2. Runs the 15 checks → a score (58, in this case — medium risk, mostly held back by no OAuth support and a moderate dependency footprint)
3. Generates a ready-to-paste config for Claude Desktop / Cursor / VS Code / Windsurf
4. Extracts its actual tool list and FAQs from the README
5. Waits for a human to confirm the AI’s draft summary before anything ships

Where it stands today

213 servers, 36 categories, all re-scraped and re-scored on a rolling basis
so a score reflects last week’s commit activity, not last year’s.

It’s open at repoai.io — MCP is the first layer, Claude
Skills and the rest of the AI-tool ecosystem are next, same trust model.

Curious what other people would want a security scorer to check for —
what’s bitten you when installing an MCP server?