SEC.gov publishes two feeds that matter a lot to anyone tracking enforcement
risk: litigation releases and administrative proceedings. Neither has an
API. Both are static web pages you’re expected to refresh by hand, and
neither one tells you the actual EDGAR-registered company behind the
respondent’s name — just whatever name the release itself used.
This post walks through how I built a small, pay-per-event Actor that turns
both feeds into a structured, delta-tracked stream — and specifically, the
three design decisions that mattered most: fingerprint-based delta tracking,
automatic EDGAR CIK resolution, and honest failure classification.
The core problem: re-scraping is wasteful, and identity resolution is manual
If you scrape SEC.gov’s litigation-release page on a schedule, the naive
approach re-pulls and re-processes the entire feed every run. Most of that
feed hasn’t changed since your last run. You end up paying (in compute, in
API costs, in your own attention re-reading output) for a mountain of
“nothing happened” noise around a handful of genuinely new records.
Worse: the release only ever names a respondent the way the SEC wrote it —
“Example Capital Management LLC.” If your downstream system needs the
EDGAR CIK to actually join this against a company you have on file, that’s
manual cross-referencing, every single time, for every single release.
Architecture
[ SEC.gov Releases ] ──┐
├──> [ Canonicalize Record ] ──> [ Fingerprint Hash ]
[ SEC.gov Proceedings ] ┘ │
▼
/ Seen Before? \
/ \
YES NO
│ │
▼ ▼
[ Drop - $0.00 Billed ] [ EDGAR CIK Match ]
│
▼
[ Parse Sanctions ]
│
▼
[ Persist New Hash ]
Enter fullscreen mode Exit fullscreen mode
1. Fingerprint-based delta tracking
Every record — release or proceeding — gets canonicalized (consistent field
ordering, whitespace normalization, date formatting) and hashed. That hash
is compared against a persisted state store from the previous run. If the
hash matches, the record is dropped before it ever reaches the billing
layer. Apify’s Pay-Per-Event model means an all-unchanged run genuinely
costs the caller $0.00 — the fingerprinting isn’t just an optimization, it’s
the actual pricing mechanism.
import requests
response = requests.post(
"https://api.apify.com/v2/acts/EDhT9Mvrdm2hzTECA/run-sync-get-dataset-items",
params={"token": "<YOUR_API_TOKEN>"},
json={"userAgent": "YourCompany [email protected]", "maxItemsPerRun": 50, "onlyNew": True},
)
records = response.json()
print(f"{len(records)} records returned")
Enter fullscreen mode Exit fullscreen mode
Run this twice in a row against an unchanged feed and the second call
returns an empty array — and bills nothing.
2. Automatic EDGAR CIK resolution
This is the part that actually took the most engineering time. Each
respondent name gets cross-referenced against EDGAR’s own filer database,
including known former company names (mergers, rebrands), and the match
comes back with a numeric confidence score rather than a blind first-result
match:
{
"record_id": "LR-26636",
"event_type": "NEW_LISTING",
"primary_respondent": "Example Capital Management LLC",
"linked_edgar_cik": "0001234567",
"edgar_match_confidence": 0.94,
"monetary_sanctions": { "sought_usd": 450000, "ordered_usd": 310000 }
}
Enter fullscreen mode Exit fullscreen mode
The confidence score matters because respondent-name matching against a
public filer database is inherently fuzzy — surfacing the score instead of
hiding it behind a boolean lets a downstream consumer decide their own
threshold for “trust this match automatically” vs. “flag for manual
review.”
3. Honest failure classification
This lesson came from a sibling project, not this one. A monitor I built
for the UK’s Health and Safety Executive register hit a real multi-day
upstream outage — HSE’s own servers were unreachable at the network level.
Without explicit handling, that outage would have surfaced as a generic
“run failed” error, indistinguishable from an actual bug in my own
scraping logic. That’s a genuinely bad failure mode for anyone depending on
the feed: you can’t tell “wait it out” from “file a bug report” from the
error alone.
So this Actor (and now the rest of the fleet) explicitly classifies a
network-level failure to reach the source as an upstream outage, tagged
distinctly from an internal parsing or logic error. It sounds like a small
thing until you’re the one staring at a red run and trying to decide
whether to open an issue.
Pricing, for context
- New enforcement release: $0.05
- Correction to a previously-delivered release (rare — these are close to immutable once published): $0.02
- Unchanged release: $0.00
Try it
curl -X POST "https://api.apify.com/v2/acts/EDhT9Mvrdm2hzTECA/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
-H "Content-Type: application/json" \
-d '{"userAgent": "YourCompany [email protected]", "maxItemsPerRun": 50, "onlyNew": true}'
Enter fullscreen mode Exit fullscreen mode
Source is on GitHub:
stefanoseggio/sec-enforcement-litigation-delta-feed.
The hosted Actor is on the Apify Store
if you’d rather not run it yourself.
If you’re solving a similar “no first-party API, entity resolution
required” problem against a different government data source, I’d like to
hear how you approached it — the delta-fingerprinting pattern here
generalizes pretty directly to most of these feeds.