Practical Web Scraping in Python: Playwright, Scrapy, and Where They Meet

If you’ve ever had to write 50 lines of boilerplate just to launch a headless browser and grab a single string of text, you know the pain. Modern front-end development loves hiding basic content behind megabytes of client-side JavaScript—but that doesn’t mean every scraper needs a full browser engine running behind it.

When building Python data pipelines, we usually face two distinct problems: fetching raw web content and interacting with dynamic DOMs. Let’s break down how Scrapy and Playwright tackle these problems, their architectural trade-offs, and how to seamlessly bridge them using scrapy-playwright.

1. Choosing the Right Mental Model

Before writing a line of code, ask yourself: Are you reading raw data, or are you executing a live web app?

Feature Scrapy Playwright Primary Role Asynchronous web crawling engine Headless browser automation framework Target Engine Pure HTTP responses / Raw HTML Real browser binaries (Chromium, Firefox, WebKit) JS Rendering ❌ None (by default) Full DOM & client JS execution Throughput ⚡ Ultra-high speed, minimal RAM 🐢 Resource-intensive, CPU/RAM bound

Rule of thumb: If the target site returns clean static HTML or standard JSON endpoints, stick with Scrapy. If the text renders dynamically after page load via heavy JS framework calls, bring in Playwright.

2. Playwright: Clean, Explicit Automation

Playwright keeps resource management straightforward through Python context managers.

Standard Synchronous Flow

Ideal for standalone CLI tools or scripts:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("[https://example.com](https://example.com)")
    print(page.title())
    browser.close()

Enter fullscreen mode Exit fullscreen mode

When execution leaves the with block, Playwright cleanly shuts down the browser process—preventing orphaned Chromium instances from eating up host memory.

Asynchronous Execution

If you’re integrating into an asyncio loop or web backend, use async_playwright:

import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto("[https://example.com](https://example.com)")
        print(await page.title())
        await browser.close()

asyncio.run(main())

Enter fullscreen mode Exit fullscreen mode

Pro Tip: Hit APIs directly with Playwright’s Network Stack

If a page exposes internal REST endpoints, don’t waste CPU cycles rendering the layout engine. You can use Playwright’s underlying request context directly:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    api_context = p.request.new_context(base_url="[https://jsonplaceholder.typicode.com](https://jsonplaceholder.typicode.com)")
    response = api_context.get("/posts/1")
    print(response.json())
    api_context.dispose()

Enter fullscreen mode Exit fullscreen mode

3. Combining Forces: scrapy-playwright

What if you need Scrapy’s link extractors, request scheduling, item pipelines, and throttling, but your pages strictly require client-side JavaScript execution?

You bridge them using scrapy-playwright.

Resolving the Event Loop Problem

Scrapy relies on Twisted, whereas Playwright runs on native `asyncio. To join them, configure Scrapy to use Twisted's AsyncioSelectorReactor and set scrapy-playwright` as the custom download handler.

import scrapy

class PlaywrightQuotesSpider(scrapy.Spider):
    name = "playwright_quotes"

    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
    }

    def start_requests(self):
        yield scrapy.Request(
            url="[https://quotes.toscrape.com/js/](https://quotes.toscrape.com/js/)",
            meta={
                "playwright": True,
                "playwright_include_page": True,
            },
        )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        await page.wait_for_selector("div.quote")
        content = await page.content()

        sel = scrapy.Selector(text=content)
        for quote in sel.css("div.quote"):
            print(quote.css("span.text::text").get())

        await page.close()

Enter fullscreen mode Exit fullscreen mode

Architectural Gotchas & Pitfalls

1. RAM Limits in Production: Running 50 concurrent Playwright browser tabs consumes gigabytes of memory compared to a few megabytes for standard Scrapy HTTP connections. Only activate meta={"playwright": True} on requests that actually require client-side JS.
2. Reactor Initialization Order: The TWISTED_REACTOR setting must be configured before Twisted imports or initializes its default reactor. Setting it in custom_settings (as above) or settings.py ensures it hooks early enough.

4. Production Decision Matrix

• Scrapy Alone: Best for large-scale, multi-domain crawls, static HTML sites, and direct API extraction.
• Playwright Alone: Best for UI end-to-end testing, automated logins/form filling, screenshots, or single-page app extractions.
• Scrapy + Playwright: Best when you need production crawling infrastructure (queues, pipelines, proxies, throttling) but face dynamic JavaScript obstacles.

What’s your go-to setup for handling heavy JavaScript rendering in Python scrapers? Do you prefer scrapy-playwright, browserless API hooks, or a custom headless setup? Drop your thoughts below!