Migrating a store to a new platform is mostly one boring, expensive problem: getting the product catalog out of the old site. The checkout, the theme, the domain — those are afternoons. The catalog is weeks.
Here is how to do it in minutes instead, and why the three obvious approaches usually don’t get you there.
Why catalog migration eats weeks
1. Retyping by hand
The default. Someone opens the old store in one tab and the new admin in another, and copies 600 products across. At a realistic 3-4 minutes per product — name, description, price, SKU, category, download the photo, re-upload it — that’s about 35 hours of work. It is also the version that silently loses data: whoever gets tired at product 400 starts skipping the specs table.
2. The CSV export that “already exists”
Every platform has an export button, so this feels solved. It isn’t:
- The export gives you their column names, not the new platform’s.
- Photos come as URLs pointing at the old CDN, which dies with the old plan.
- Variants, bundles and specs get flattened into one unparseable text blob.
- Legacy or custom-built stores (a huge share of small B2B sites) have no export at all.
You end up writing a transform script anyway — and you can only start once you have someone’s admin credentials, which for an agency migrating a client is often the long pole.
3. A scraper with CSS selectors
The engineer’s answer, and it works — for exactly one site. You open DevTools, find that the price lives in .product-info__price > span, write the selector, and it holds until the store’s theme updates. Multiply by every client you migrate and you are maintaining a per-site template library. Selector-based scrapers don’t generalize; that’s their whole problem.
The AI approach: no selectors at all
Instead of teaching a script where the price is, you let a model read the page and tell you what’s on it. The crawler is generic — follow links, find category pages, follow pagination, collect candidate product URLs — and the extraction is a prompt.
The non-obvious part is not extraction. It’s rejection. E-commerce pages lie: a category page carries a name, a price and a photo, exactly like a product page. So the single most important instruction in the prompt is that the model must answer null when the page is a listing, not a product. Teaching the AI to say “this isn’t a product” mattered more than teaching it to extract.
Step by step, with curl
I packaged the engine as an API. Subscribe on RapidAPI (free tier, no card), grab your key, and it’s two calls: submit a job, poll it.
1. Submit the store URL
curl -X POST 'https://ai-product-catalog-extractor.p.rapidapi.com/v1/extract' \
-H 'x-rapidapi-key: YOUR_KEY' \
-H 'x-rapidapi-host: ai-product-catalog-extractor.p.rapidapi.com' \
-H 'content-type: application/json' \
-d '{
"url": "https://the-old-store.com/",
"limites": { "max_paginas": 200 }
}'
Enter fullscreen mode Exit fullscreen mode
You get a 202 back:
{ "job_id": "9f2c...", "poll": "/v1/jobs/9f2c..." }
Enter fullscreen mode Exit fullscreen mode
2. Poll until it’s done
curl 'https://ai-product-catalog-extractor.p.rapidapi.com/v1/jobs/9f2c...' \
-H 'x-rapidapi-key: YOUR_KEY' \
-H 'x-rapidapi-host: ai-product-catalog-extractor.p.rapidapi.com'
Enter fullscreen mode Exit fullscreen mode
While it runs you get live progress — fase (phase), paginas_visitadas, itens_extraidos — so you can render a progress bar instead of a spinner.
3. Read the catalog
{
"status": "done",
"result": {
"itens": [
{
"nome": "Rebite Repuxo Alumínio 4,8 x 12mm",
"marca": "CRV",
"categoria": "Rebites",
"sku": "RA-4812",
"preco": 42.9,
"descricao": "Rebite de repuxo em alumínio com haste de aço...",
"foto": "https://old-store.com/img/ra-4812.jpg",
"_url": "https://old-store.com/produtos/rebite-repuxo-4812"
}
],
"total_candidatos": 126,
"truncado": false,
"usage": { "tokens_entrada": 113559, "chamadas_ia": 60 }
}
}
Enter fullscreen mode Exit fullscreen mode
About those Portuguese field names. The engine was born inside a Brazilian ERP, so the default preset ships nome, preco, foto, descricao. That’s cosmetic, not structural — you can send your own schema and get whatever keys your target platform wants:
"schema": [
{ "key": "title", "tipo": "texto", "label": "product name" },
{ "key": "price", "tipo": "number" },
{ "key": "image", "tipo": "foto" }
]
Enter fullscreen mode Exit fullscreen mode
Same engine, keys ready to POST straight into Shopify or WooCommerce.
Real numbers
A cold run against a Brazilian industrial-tools store, no configuration, no selectors:
160 pages crawled → 126 candidates → 50 clean products in 72 seconds. Zero duplicates, zero category pages leaking into the results, ~113k input tokens (a few cents of model cost).
Compare that with 35 hours of retyping.
Honest limitations
- SPA storefronts don’t work yet. If the products only exist after JavaScript runs (VTEX, headless Shopify, client-side Next.js), you’ll get zero. Static HTML only, for now.
- One page = one product is assumed. Stores where SKUs exist only inside a filterable listing, with no individual URL, return nothing.
- robots.txt is respected by default. There’s an owner-authorization flag for when it’s your own store.
- Crawling is rate-limited per host on purpose, so a large catalog takes minutes, not seconds. Pointing at the listing page instead of the homepage speeds things up a lot.
Try it on your own migration
- API listing (free tier, 50 calls/month): AI Product Catalog Extractor on RapidAPI
- Live demo of a real extraction: 72.60.244.148:3024
- How it was built, and the three cleanup guards that came from production bugs: I built an AI API that turns any online store URL into a structured product catalog
If you point it at a store and it breaks, tell me which one — the edge cases are the roadmap.