엔트로피 게이팅을 사용하여 DLP 오탐을 80% 줄인 방법

We’ve been building Spidercob, an enterprise DLP platform. The hardest problem was never detecting sensitive data it was not detecting things that aren’t sensitive.

Regex-based scanners are easy to write. A pattern for AWS access keys takes five minutes. The problem is that same pattern fires on any 20-character uppercase string in your codebase test fixtures, documentation examples, README placeholders, all flagged as critical.

After six months of tuning in production we extracted our detection engine into a standalone library:
dlp-patterns. Here’s what actually works.

## The Three Layers

### 1. Validators

The first thing we added was format validation after the regex match. A regex is a necessary condition, not a sufficient one.

Credit cards – every match runs through Luhn’s algorithm. A transposed digit fails Luhn. This alone cuts credit card false positives by roughly 90%.

python
  def _luhn_check(number: str) -> bool:
      digits = [int(d) for d in number if d.isdigit()]
      odd_digits = digits[-1::-2]
      even_digits = digits[-2::-2]
      total = sum(odd_digits)
      for d in even_digits:
          total += sum(divmod(d * 2, 10))
      return total % 10 == 0

  SSNs — the IRS publishes rules about which area codes are invalid. 000, 666, and 900-999 are all unassigned. We reject those before surfacing a finding.

  JWTs — we base64-decode the header and payload and verify they parse as valid JSON with expected fields (alg, typ). A random string that happens to match the JWT pattern fails this check immediately.

  2. Entropy Gating

  This is the biggest lever we found. Low-entropy strings are almost never real secrets.

  Shannon entropy measures how random a string is. A real AWS secret key looks like noise. A test placeholder like AKIAIOSFODNN7EXAMPLE —
  ironically AWS's own documentation example — has lower entropy because it contains recognisable English words.

  We use a sliding window approach:

  import math

  def _shannon_entropy(s: str) -> float:
      if not s:
          return 0.0
      freq = {}
      for c in s:
          freq[c] = freq.get(c, 0) + 1
      length = len(s)
      return -sum(
          (count / length) * math.log2(count / length)
          for count in freq.values()
      )

  def _passes_entropy_gate(value: str, min_entropy: float = 3.5) -> bool:
      if _shannon_entropy(value) < min_entropy:
          return False
      # Sliding window catches strings like "aaabbbccc" that fool overall entropy
      window = 8
      for i in range(len(value) - window + 1):
          chunk = value[i:i + window]
          if len(set(chunk)) < 4:
              return False
      return True

  This single check eliminates the vast majority of generic secret false positives. Repeated characters, dictionary words, sequential patterns —
  all rejected without touching a single regex.

  3. Context Scoring

  The last layer doesn't reject findings outright — it scores them.

  Every finding gets a context_score between 0.0 and 1.0 based on the words surrounding it. We scan ±100 characters around each match:

  Boost keywords — score goes up: production, prod, secret, credential, deploy, live, api_key

  Penalty keywords — score goes down: example, placeholder, test, sample, dummy, fake, replace

  BOOST_KEYWORDS = {"production", "prod", "secret", "credential", "deploy", "live"}
  PENALTY_KEYWORDS = {"example", "placeholder", "test", "sample", "dummy", "fake"}

  def _context_score(text: str, match_start: int, match_end: int) -> float:
      window_start = max(0, match_start - 100)
      window_end = min(len(text), match_end + 100)
      context = text[window_start:window_end].lower()
      words = set(context.split())

      score = 0.5  # neutral baseline
      score += len(words & BOOST_KEYWORDS) * 0.15
      score -= len(words & PENALTY_KEYWORDS) * 0.20
      return max(0.0, min(1.0, score))

  You can threshold on context_score in your pipeline. In CI, only block on findings with context_score > 0.7. In a compliance scan, report
  everything but prioritise high-score findings.

  Required Context Keywords

  Some patterns are too ambiguous to fire without corroborating evidence.

  ICD-10 codes like E11.9 are a good example. The pattern [A-Z]\d{2}\.\d{1,4} matches thousands of strings that aren't medical codes. We only fire
   the ICD-10 pattern when medical keywords appear nearby:

  diagnosis, icd, icd-10, condition, disease, disorder,
  patient, clinical, medical, treatment

  Without one of those words in context the pattern stays silent. This approach works well for any structurally ambiguous pattern — Telegram bot
  tokens, NPI numbers, DEA registration numbers.

  Using It

  import dlp_patterns

  result = dlp_patterns.scan(text)

  result.has_findings          # bool
  result.highest_severity      # "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | None
  result.critical              # list[Finding]

  f = result.critical[0]
  f.type                       # "aws_access_key"
  f.value                      # masked value
  f.context_score              # 0.0 – 1.0
  f.context_keywords_found     # ["production", "deploy"]

  # Redact in one line
  clean = dlp_patterns.redact(text)

  # Secrets only — skip PII, faster for CI
  result = dlp_patterns.scan(code, secrets_only=True)

  CLI for pipelines:

  # Fail CI on CRITICAL findings
  dlp-scan -secrets-only src/

  # Redact logs before shipping to external services
  cat app.log | dlp-scan -redact > safe.log

  # JSON output for SIEM integration
  dlp-scan -json document.txt | tee dlp-report.json

  GitHub Actions — use the official action:

  - name: DLP Secret Scan
    uses: spidercob/dlp-scan-action@v1
    with:
      secrets-only: 'true'
      fail-on: 'critical'

  50+ Pattern Categories

  ┌────────────────┬────────────────────────────────────────────────────────────────────────┐
  │    Category    │                                Patterns                                │
  ├────────────────┼────────────────────────────────────────────────────────────────────────┤
  │ Secrets        │ AWS, GitHub PATs, Stripe, SendGrid, Slack, Twilio, Discord, Google API │
  ├────────────────┼────────────────────────────────────────────────────────────────────────┤
  │ Infrastructure │ DB connection strings, hardcoded passwords, Docker registry auth       │
  ├────────────────┼────────────────────────────────────────────────────────────────────────┤
  │ Crypto keys    │ RSA / EC / SSH / PGP private keys, X.509 certs                         │
  ├────────────────┼────────────────────────────────────────────────────────────────────────┤
  │ PII            │ SSN, credit cards, email, US phone, passport, date of birth            │
  ├────────────────┼────────────────────────────────────────────────────────────────────────┤
  │ Healthcare     │ Medical record numbers, ICD-10 codes, NPI, DEA numbers                 │
  ├────────────────┼────────────────────────────────────────────────────────────────────────┤
  │ Crypto wallets │ Bitcoin addresses, Ethereum addresses                                  │
  └────────────────┴────────────────────────────────────────────────────────────────────────┘

  Install

  pip install dlp-patterns

  Zero dependencies. Python 3.9+. Apache 2.0.

  Source: https://github.com/SpiderCob/dlp-patterns

  GitHub Action: https://github.com/SpiderCob/dlp-scan-action

  If you're scanning at scale and need dashboards, audit logs, ICAP proxy integration, or Gmail/Slack connectors — that's what
  https://spidercob.com is built for.

Enter fullscreen mode Exit fullscreen mode