Disclosure: I work on Mockzilla, which competes with Stoplight Prism.
Everything below was measured on my laptop, and there is a link to the longer
comparison at the end.
Prism is the default answer to “I need to mock this OpenAPI spec”, and for most
specs it is the right one. Install it, point it at a document, every operation
in the file answers.
npm install -g @stoplight/prism-cli
prism mock openapi.yml
Enter fullscreen mode Exit fullscreen mode
Then I gave it Stripe.
The run
Stripe publishes its API as an OpenAPI document. The copy I used is 6.0 MB, 414
paths and 587 operations. Large, but not unusual: GitHub’s is 8.8 MB.
$ prism mock stripe.yml
[CLI] … awaiting Starting Prism…
Enter fullscreen mode Exit fullscreen mode
Nothing followed it. Thirty minutes on it was still that one line, the
process pinned at 100% of a core with 216 MiB resident and nothing listening on
port 4010. That is where I stopped it.
Handing it JSON instead does not help. The same document as stripe.json, same
machine: no listener after fifteen minutes, at 274 MiB. There is no flag for
this and nothing was misconfigured.
Where the time goes
Not parsing. Python’s YAML parser loads the same file in 5.3 seconds.
It goes on resolution. Stripe’s document carries 1,286 component schemas with
3,700 $refs pointing into them, and Prism walks all of that before it binds a
port. A spec with a few hundred inline schemas never shows you this.
The file itself is servable. The mock server I work on opens a port on the same
document in 195 ms and answers the first request at 364 ms.
What it costs on a spec that does work
Twilio Verify: 29 paths, 53 operations, 173 KB of YAML. Installed natively,
three runs.
Half a second and 200 MB is nothing on a laptop. A container that boots and
dies on every CI job pays it every time.
One process, one document
“A single Prism instance serves one OpenAPI document.” Prism’s documentation
says so, and what it recommends for a second document is a second process on a
second port with a reverse proxy in front of both:
services:
proxy:
image: caddy
ports:
- '8080:80'
depends_on:
- prism_1
- prism_2
prism_1:
image: stoplight/prism:4
command: mock -p 4010 --host 0.0.0.0 petstore-v2.yaml
prism_2:
image: stoplight/prism:4
command: mock -p 4010 --host 0.0.0.0 petstore-v3.yaml
Enter fullscreen mode Exit fullscreen mode
Most test suites talk to more than one vendor. Three of them is three Node
processes, three ports, a Caddyfile and something to supervise the lot, on
every developer machine and in every CI job. Each process holds its own copy of
its own document, so the 191 to 199 MiB above is per API rather than per
machine.
What Prism does better
Three things, and if you need the first one this post ends here.
Forcing a status code, per request. Prefer: code=404 on any call, or
?__code=404 if a header is awkward, and Prism serves that response out of the
document. Ask for a code the document does not declare and you still get the
status, with a Prism error body saying so. What I run has no per-request
equivalent.
Enforcing the document’s security. Twilio Verify declares HTTP basic auth,
so Prism answers 401 with a WWW-Authenticate header until you send
credentials. Prism’s behaviour is the contract-accurate one, and it catches a
client that forgot to send a token.
Older and stranger documents. OpenAPI 2.0, the version still called
Swagger, and Postman Collections. If your document has not been converted yet,
that decides it.
If your spec is the big kind
A mock server that reads the document at start-up and resolves all of it has a
ceiling, and Stripe’s spec is above it. Worth knowing before you put prism in a CI job and find out from a timeout.
mock
I wrote up the full comparison with both tools running the same specs, the
generated data either side, the per-request controls, and the prices:
Stoplight Prism alternative: an OpenAPI mock server with a URL