How to Block Resources in Playwright and Python for Efficient Web Scraping

When scraping websites using tools like Playwright and Python, you often don‘t need to load every single resource on the page. Things like images, fonts, stylesheets, and scripts may be irrelevant to the data you‘re trying to extract. Loading these resources takes up extra bandwidth and slows down your scraper.

Content Navigation show

Fortunately, Playwright provides the ability to intercept and block network requests for selected resources. By preventing unnecessary resources from loading, you can make your web scraper faster and more efficient. In this guide, we‘ll walk through how to block resources by type, URL, or URL pattern using Playwright in Python. We‘ll cover examples and best practices to help you optimize your scraping projects.

Overview of Playwright for Web Scraping

Playwright is a powerful browser automation library that allows you to control headless or real browsers like Chrome, Firefox, and Safari with code. Playwright supports multiple languages including Python, Node.js, Java, and .NET.

Some key features that make Playwright very useful for web scraping:

  • Can run scripts and handle dynamic page content
  • Automatically waits for elements and page loads
  • Emulates mobile devices and geolocation
  • Can generate PDFs and capture screenshots
  • Ability to intercept and mock network requests

Here‘s a quick example of using Playwright to scrape web page data in Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com")
    print(page.title())
    browser.close()

This script launches a headless Chrome browser, navigates to a URL, prints the page title, and then closes the browser. Playwright can do much more advanced automation and data extraction as well.

Why Block Resources?

Loading resources like large images, fonts, and scripts takes time and bandwidth. If your scraper is just pulling text data or you‘re crawling thousands of pages, this overhead can really add up.

There are a few key reasons you may want to block selected resources when scraping with Playwright:

  1. Improve scraping speed – By preventing images and other large files from loading, your scraper can navigate through pages much faster. This allows you to crawl more pages in less time.

  2. Save bandwidth – Blocking resources means less data transferred per page. This can significantly reduce your bandwidth consumption, which is especially important if you‘re running scrapers in the cloud or on metered connections.

  3. Avoid bot detection – Some websites use techniques like monitoring resource loading patterns to detect and block bots. By acting more like a real user that doesn‘t necessarily load all resources, blocking some requests may help your scraper fly under the radar.

  4. Isolate page content – When you‘re just trying to pull text, blocking images and styling can make it easier to work with the core page HTML without extraneous elements getting in the way.

Of course, blocking resources indiscriminately can cause issues on some sites. For complex SPAs and pages that lazy-load critical content, you may need to experiment to find the optimal blocking strategy. Next, we‘ll look at how Playwright lets you control which resources to allow or block.

Blocking Resources by Type

Playwright‘s Page object provides a route method that lets you intercept and handle network requests. You can set up routing to block selected resource types using the Request.resourceType property.

Here‘s an example that blocks all images from loading:

from playwright.sync_api import sync_playwright

def block_images(route, request):
    if request.resource_type == "image":
        route.abort()
    else:
        route.continue_()

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.route("**/*", block_images)
    page.goto("https://example.com")
    print(page.title())
    browser.close()

The page.route() method takes a URL pattern and a handler function. The pattern **/* matches all requests. The block_images function checks the request.resource_type – if it‘s an image, the request is aborted. Otherwise it‘s continued using route.continue_().

Some of the available resource types you can filter on:

  • document
  • stylesheet
  • image
  • media
  • font
  • script
  • texttrack
  • xhr
  • fetch
  • other

So to block scripts for example, just check for request.resource_type == "script". You can also combine multiple types:

def block_resources(route, request):
    if request.resource_type in ("image", "font", "stylesheet"):
        route.abort()
    else:
        route.continue_()

This will block images, fonts, and stylesheets while allowing other content. Blocking stylesheets is useful for keeping just the page text content.

You can also choose to provide alternative "mock" responses instead of just aborting requests. The route.fulfill() method lets you return custom response data:

def mock_images(route, request):
    if request.resource_type == "image":
        route.fulfill(body="<svg>Mock</svg>", content_type="image/svg+xml")
    else:
        route.continue_()

Now instead of a broken image, a dummy inline SVG will be returned for all image requests. This could be used to supply placeholder images while still avoiding loading real images.

Blocking Resources by URL

What if you need more fine-grained control than just blocking entire resource types? Playwright also supports intercepting requests for specific URLs or URL patterns.

To block an exact URL:

page.route("https://example.com/annoying-popup.js", lambda route: route.abort())

This will intercept and abort requests for that specific URL on the page. Wildcards and regex patterns allow more flexible matching:

# Block a subdirectory
page.route("**/ads/*", lambda route: route.abort())

# Block a domain and all subdomains
page.route("*.google-analytics.com/**", lambda route: route.abort())  

# Block using regex
page.route(re.compile(r"\.jpg$"), lambda route: route.abort())

The first example blocks all requests to the /ads/ subdirectory. The second blocks an entire domain including subdomains. The last one uses a regex pattern to match any URL ending in .jpg.

You can get pretty creative with pattern matching to surgically block scripts, trackers, and other undesirable resources. Blocking using URLs is often more robust than resource types, since sites can disguise 3rd-party scripts as fetch or xhr resource types.

Here‘s a full example that shows blocking resources while scraping a page:

import re
from playwright.sync_api import sync_playwright

def block_resources(route, request):
    resource_type = request.resource_type
    url = request.url

    if resource_type in ("image", "font", "stylesheet"):
        route.abort()
    elif re.search(r"(ads|tracking|analytics)", url):
        route.abort()
    else:
        route.continue_()

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.route("**/*", block_resources)
    page.goto("https://example.com")

    # Scrape page data
    title = page.title()
    description = page.query_selector(‘meta[name="description"]‘).get_attribute("content")

    print(f"Title: {title}")
    print(f"Description: {description}")

    browser.close()

This scraper blocks images, fonts, and stylesheets, as well as any URLs containing "ads", "tracking", or "analytics". The rest of the content is allowed to load normally for scraping the title and description. You can see how blocking the extraneous resources helps zero in on the meaningful data.

Best Practices and Gotchas

Here are a few tips and things to watch out for when blocking resources in Playwright:

  • Start by blocking conservatively (just larger resources like images) and add more blocked types as needed
  • Avoid blocking document and xhr/fetch resources unless you‘re sure the page doesn‘t need them
  • If a page looks broken after blocking, try allowing stylesheets and fonts
  • Unblock resources if you encounter issues with page functions, forms, logins, etc.
  • Regularly test and update blocked URL patterns, as site structures can change
  • Consider blocking 3rd-party scripts and trackers for scraping speed and stealth

It‘s also a good idea to compare different scraping approaches to find the optimal balance of performance and data quality for your needs. Experiment with blocking settings and check scraped data to see what works best.

Blocking Resources in Other Scraping Tools

Playwright is not the only scraping tool that supports blocking resources, though its implementation is quite flexible and easy to use. Here‘s a quick comparison to some other common Python scraping libraries:

  • Scrapy: Provides a MEDIA_ALLOW_REDIRECTS setting and NoImagesPipeline class to avoid downloading images. Doesn‘t have built-in support for blocking other resource types.
  • Requests: Doesn‘t execute JS, so many resource requests won‘t be made by default. Can block domains using a before_request hook on the Session object.
  • Selenium: Can block images with prefs["profile.managed_default_content_settings.images"] = 2 in Chrome options. No built-in way to block urls, but can use request interception in Chrome DevTools Protocol.
  • HTTPX: Like Requests, doesn‘t load most resources out of the box as a headless client. Can perform some basic domain blocking with a request event hook.

For most scraping needs, Playwright‘s combination of full browser rendering and granular request interception provides the best of both worlds. But depending on your requirements, tools like Scrapy or Requests can also get the job done with some resource blocking.

Wrap Up

Blocking resources is one of the most effective ways to speed up and streamline your web scraping pipelines. Playwright makes it easy to block requests by resource type, URL, or regex pattern in Python. By preventing images, scripts, and other unnecessary resources from loading, your scraper can run faster, consume less bandwidth, and be more resistant to bot detection.

The techniques covered in this guide provide a flexible arsenal for optimizing your Playwright scrapers. You can mock out distracting page elements, block tracking scripts and analytics, and hone in on the content that matters. Of course, every website is different, so be sure to test thoroughly and make adjustments as needed.

We encourage you to experiment with resource blocking in your own scraping projects. And if you‘re new to Playwright, check out our full tutorial for a deep dive on browser automation and web scraping in Python. Happy scraping!