Thanks! A few decisions do most of the work:
Bootstrap is a keyset-paginated snapshot with prefetch — it pages through the source by primary key (never OFFSET), and while one chunk is being written to the sink, the next is already being fetched. The engine never holds more than a couple of chunks in memory, which is why RSS stays flat (~250MB) no matter how big the dataset is.
Deterministic document ids make everything else legal. Every doc maps to the same id every time, so a retry is an overwrite, not a duplicate. That’s what lets the dispatcher keep up to 16 bulk requests in flight against OpenSearch (with external versioning so reordering can’t corrupt anything) instead of writing one batch at a time.
Docs pass through mostly as raw JSON — no deserialize/reserialize round-trip on the hot path.
OpenSearch is configured for ingest during bootstrap — refresh disabled, no replicas, one final refresh before the count check.
Graph sources stream instead of paginate. For Neo4j, the bootstrap runs one streamed Bolt query over the graph rather than paging with LIMIT/ORDER BY — the driver pulls results lazily in fetch-size batches, so there’s no per-page re-planning or sorting on the server, memory stays bounded, and scan cost is strictly O(n) no matter how large the graph gets. That’s how a 50M-node graph bootstraps in the same ~6-minute class as the relational sources.
i love if you can star to support the project
github.com/ventstream/ventstream
please add an issue if you notice anything off