We Planted 10 Vulnerabilities to Test Free Semgrep. It Reported 3.

Before you read: we are not security experts. We are a small team learning security tooling by testing it directly and writing down what happened. This is a field note, not a best-practices guide. If something here is wrong, corrections in the comments are welcome.

TL;DR

We put 10 common vulnerabilities into a React and TypeScript app on purpose, then ran the free, zero-config Semgrep (--config=auto) against them. It reported 3 of the 10. The gap is not a defect in Semgrep. It reflects what free static analysis is built to find and what it is not. The details are below.

[IMAGE PLACEHOLDER, COVER] A cover image goes here. The scorecard graphic or a screenshot of the GitHub Actions run both work.

Why we ran this

TL;DR: We wanted a measured answer to one question. On its own, how much does a free SAST scan actually find?

“Add SAST to your CI” is common advice. The part that rarely gets measured is what you get from it when you pay nothing and write no custom rules. Instead of guessing, we set up a small test:

• A Security Operations Dashboard built with React 18, TypeScript, and Vite.
• A folder, src/security-test-cases/, holding 10 deliberately vulnerable files, one per weakness.
• A GitHub Actions workflow that runs semgrep scan --config=auto on every push.

Then we read the CI logs and recorded the results.

How Semgrep works

TL;DR: Semgrep parses your code into a tree (an AST) and matches rule patterns against that tree. It does not run your program. If no rule describes a given pattern, that pattern is not reported.

The same idea as a text diagram:

  Source code  ->  Parser  ->   AST      ->  Match rules  ->  Findings
  (.ts/.tsx)                  (a tree of       (~1074           (file + line
                               your code)       patterns)         + rule id)

Enter fullscreen mode Exit fullscreen mode

A few things follow from this design:

• It is fast and cheap to run. There is no compile or execution step, so a full repo scans in seconds, which suits CI.
• It handles bad code that is present. A call like crypto.createHash('md5') has a stable shape that a rule can match.
• It struggles with correct code that is absent. The missing presence of an authorization check is hard to express as a pattern.

The 10 cases we planted

TL;DR: One file per weakness, each mapped to its CWE identifier. All are isolated and never imported by real application code.

# Vulnerability CWE Risk in one line 1 Hardcoded credential CWE-798 Secrets in git history are hard to rotate and are exposed to anyone with repo access 2 IDOR CWE-639 Trusting a user-supplied ID lets one user read another user’s data 3 Sensitive info leak (logs or localStorage) CWE-532/312 Tokens in logs or localStorage are easy to read 4 Broken crypto (MD5) CWE-327 MD5 is unsafe for password or integrity use 5 XSS via dangerouslySetInnerHTML CWE-79 Rendering raw user HTML lets a script run in another user’s session 6 SQL or NoSQL injection CWE-89 String-concatenated queries let an attacker rewrite the query 7 Insecure deserialization or unsafe eval CWE-502 Evaluating untrusted input allows code execution 8 CSRF (no token) CWE-352 Cookie-only state changes can be triggered by another site 9 Open redirect CWE-601 Redirecting to an unchecked URL enables convincing phishing 10 Improper error handling CWE-209 Raw stack traces reveal internal paths and logic

The setup did not work on the first try

TL;DR: The pipeline broke several times before it produced results. Each failure had a specific cause. The sequence is below.

Attempt 1  Failed. We used a flag that does not exist, `--soft-fail`.
           CI stopped with "unknown option". Lesson: check the CLI reference, not memory.

Attempt 2  Failed. The SARIF upload hit "Resource not accessible by integration".
           Cause: missing `permissions: security-events: write`. Tokens are read-only by default.

Attempt 3  Partial. On a private repo, GitHub code scanning (Advanced Security) was unavailable.
           We made the upload non-blocking. The Security tab is not free on private repos.

Attempt 4  Not useful. `--sarif --output=file` suppressed the readable report.
           The log showed only a count, no paths. We split it into two steps.

Attempt 5  Worked. Findings appeared in the log with path, line, and rule id.

Enter fullscreen mode Exit fullscreen mode

Most of our time went into configuration rather than security: flags, permissions, and output formats. That is worth planning for.

Results

TL;DR: The first round, with generic mock code, reported 1 of 10. After we rewrote the fixtures to resemble real framework code, it reported 3 of 10. The remaining 7 stayed unreported across six different rule packs.

# Vulnerability Round 1 (generic) Round 2 (realistic) 4 Broken crypto (MD5) ✅ ✅ 7 Unsafe eval() ❌ ✅ 9 Open redirect ❌ ✅ 1 Hardcoded secret ❌ ❌ 2 IDOR ❌ ❌ 3 Sensitive info leak ❌ ❌ 5 XSS ❌ ❌ 6 SQL or NoSQL injection ❌ ❌ 8 CSRF ❌ ❌ 10 Stack-trace leak ❌ ❌

The unplanned finding

TL;DR: The same scan reported 3 real issues we did not plant, which is useful evidence that the tool works on real code.

• A mutable GitHub Actions tag (@v4 rather than a pinned SHA), which is a supply-chain risk.
• An incomplete-sanitization issue in our actual metricsApi.ts.
• A ReDoS risk from a non-literal RegExp in our actual emlParser.ts.

The 7 misses are not a sign of a broken scanner. On production code, it reported genuine problems.

Why 7 were not reported

TL;DR: Three structural reasons, none of them a bug. Pattern matching has limits.

It matches shapes, not runtime behavior

A static matcher cannot follow a value from an HTTP request through several functions to a dangerous call unless a rule already describes that flow. Deeper cross-file taint tracking is mostly a paid feature.

Missing-check bugs are hard for generic rules

IDOR and CSRF are not cases of bad code being present. They are cases of expected code being absent, such as a missing ownership check or a missing CSRF token. A generic community rule has no way to know what your application’s auth is supposed to look like.

Secrets detection is a separate tool

Finding hardcoded keys relies on entropy analysis and provider-specific patterns, closer to TruffleHog or GitGuardian than to AST matching. Our AWS and Azure style placeholders were not reported even by the dedicated secrets pack, because that capability is a different product.

What static SAST handles well, and what it does not

Where static SAST does well Where it does not Known dangerous functions such as eval and md5 Business-logic flaws such as IDOR and broken access control Risky configuration such as mutable CI tags Cases where a required check is simply missing Syntactic anti-patterns Data flow across multiple files, without paid taint tracking Fast, cheap, consistent at scale Secrets, which need a dedicated scanner

Is it still worth running? Yes, as a first layer

TL;DR: A clean Semgrep run means no known pattern matched. It does not mean there are no vulnerabilities. Use it as the fast first filter, not the only control.

What we would suggest for a small team:

1. Keep Semgrep OSS in CI. It is free and reported 3 real issues for us, so there is no reason to remove it.
2. Do not read “0 findings” as “secure”. It means no generic pattern matched.
3. Write a small number of custom rules against your own auth helpers, for example flagging any /api/ handler that does not call requireOwnership(). A handful of targeted rules will find more of your IDOR and CSRF cases than any generic pack.
4. Add a dedicated secrets scanner. Do not rely on --config=auto for keys.
5. Keep code review in the loop. A reviewer asking whether an endpoint checks ownership catches what a free static tool will not.

What we took away from it

TL;DR: The value of a tool comes from knowing where it stops.

• A green check is a starting point, not a guarantee.
• Much of the work is configuration, not analysis. Plan for that.
• Free SAST, custom rules, a secrets scanner, and code review work together. No single tool covers all of it.
• Measuring your own coverage, such as this 3 of 10, is more useful than a vendor’s claim.

If you are early in learning security, a small test like this is worth building. Testing the tool directly taught us more in a day than reading about it would have.

We are still learning, so corrections and additions in the comments are welcome.