How to scrape JavaScript-rendered websites with Python

To scrape a JavaScript-rendered website with Python, load it in a browser, wait for its content, and extract the resulting DOM. Playwright is a practical way to do that locally.

A normal HTTP request can return the page’s initial HTML before the application has created its product cards, quotes, or report rows. Parsing that response gives you the initial document. A browser executes the scripts and gives you access to the updated page.

This tutorial uses Quotes to Scrape’s JavaScript demo, a practice website. We will save structured quote data, the rendered HTML, and a screenshot. For screenshot-specific settings, see the existing Python website screenshot guide.

Choose how to retrieve the content

Before launching a browser, check where the data comes from:

A page can contain its data inside a script even when the visible cards are created later. Browser rendering is useful when you want to work with the displayed elements or capture an image of them.

Install Playwright for Python

Create a virtual environment and install Playwright and Chromium:

1
python -m venv .venv
2
source .venv/bin/activate
3
python -m pip install playwright
4
python -m playwright install chromium

On Windows, activate the environment with .venv\Scripts\activate instead of the source command.

Scrape the rendered page and save the results

Save this script as scrape-quotes.py:

1
import json
2
from pathlib import Path
3

4
from playwright.sync_api import sync_playwright
5

6
URL = "https://quotes.toscrape.com/js/"
7

8
with sync_playwright() as p:
9
    browser = p.chromium.launch()
10
    try:
11
        page = browser.new_page(viewport={"width": 1280, "height": 800})
12
        response = page.goto(
13
            URL, wait_until="domcontentloaded", timeout=30_000
14
        )
15
        if response is not None and response.status >= 400:
16
            raise RuntimeError(f"Page returned HTTP {response.status}")
17

18
        page.locator(".quote .text").first.wait_for(
19
            state="visible", timeout=20_000
20
        )
21

22
        quotes = []
23
        for card in page.locator(".quote").all():
24
            quotes.append({
25
                "text": card.locator(".text").inner_text().strip(),
26
                "author": card.locator(".author").inner_text().strip(),
27
                "tags": [
28
                    tag.strip()
29
                    for tag in card.locator(".tag").all_text_contents()
30
                ],
31
            })
32

33
        if not quotes:
34
            raise RuntimeError("No quotes found; check the page and selectors")
35

36
        Path("quotes.json").write_text(
37
            json.dumps(quotes, ensure_ascii=False, indent=2),
38
            encoding="utf-8",
39
        )
40
        Path("rendered.html").write_text(page.content(), encoding="utf-8")
41
        page.screenshot(path="quotes.png", full_page=True)
42
        print(f"Saved {len(quotes)} quotes, rendered.html, and quotes.png")
43
    finally:
44
        browser.close()

Run it with:

1
python scrape-quotes.py

The output gives you three views of the same page visit:

• quotes.json contains the fields your application can process.
• rendered.html contains the document after the readiness check.
• quotes.png shows what the browser displayed at capture time.

The selectors match the practice site. Replace them when working with your own page. The screenshot is particularly useful when debugging an unexpected empty result or checking whether the extracted data came from the right page state.

Wait for the data, not just navigation

The example waits for the first visible quote text. That is a useful signal for this demo, which inserts its quote list together. It does not prove that every item in a different application has finished loading.

For a report that streams rows progressively, wait for its completion marker. If you control the page, adding an attribute such as data-ready="true" makes the contract explicit:

1
page.locator('.report[data-ready="true"]').wait_for(
2
    state="visible", timeout=20_000
3
)

This is a replacement selector for a page with that marker; it is not part of the Quotes to Scrape demo.

Calling locator.all() reads the current matching elements. Choose your readiness condition before taking that snapshot. A fixed sleep can be too short on a slow page and waste time on a fast one. Network silence also does not establish that a particular chart or list is complete.

See Playwright’s locator waiting documentation and our guide to waiting for a page to load for more examples.

Parse rendered HTML with Beautiful Soup

If you already have parsing code built around Beautiful Soup, keep it. Render the page first and pass its HTML to the parser.

Install Beautiful Soup:

1
python -m pip install beautifulsoup4

The first script saved rendered.html. You can parse that file in a separate step:

1
from pathlib import Path
2
from bs4 import BeautifulSoup
3

4
html = Path("rendered.html").read_text(encoding="utf-8")
5
soup = BeautifulSoup(html, "html.parser")
6

7
for card in soup.select(".quote"):
8
    text = card.select_one(".text")
9
    author = card.select_one(".author")
10
    if text is not None and author is not None:
11
        print(author.get_text(strip=True), text.get_text(strip=True))

Beautiful Soup parses the supplied HTML; it does not launch the browser or run the page’s scripts. That makes the rendering step essential when your chosen fields are missing from the initial document.

The saved HTML is a document snapshot. It does not package every external image, stylesheet, or script into a portable website archive.

Get rendered HTML through ScreenshotOne

For an application that needs rendered content without running a local browser, ScreenshotOne supports format=html. You can feed that response into the same parser.

Install the HTTP client and set your API access key:

1
python -m pip install requests beautifulsoup4
2
export SCREENSHOTONE_ACCESS_KEY="your_access_key"

Save this as scrape-with-api.py:

1
import os
2
from pathlib import Path
3

4
import requests
5
from bs4 import BeautifulSoup
6

7
response = requests.get(
8
    "https://api.screenshotone.com/take",
9
    headers={"X-Access-Key": os.environ["SCREENSHOTONE_ACCESS_KEY"]},
10
    params={
11
        "url": "https://quotes.toscrape.com/js/",
12
        "format": "html",
13
        "wait_for_selector": ".quote .text",
14
    },
15
    timeout=120,
16
)
17
response.raise_for_status()
18
html = response.content.decode("utf-8")
19
Path("rendered.html").write_text(html, encoding="utf-8")
20

21
soup = BeautifulSoup(html, "html.parser")
22
cards = soup.select(".quote")
23
if not cards:
24
    raise RuntimeError("No quotes found; inspect rendered.html")
25
for card in cards:
26
    text = card.select_one(".text")
27
    if text is not None:
28
        print(text.get_text(strip=True))

Use the access key from your ScreenshotOne account. Run this as a server-side script, where the key is available in the environment.

ScreenshotOne’s wait_for_selector waits for a matching element to exist in the DOM. It does not require visibility or guarantee that a whole collection is complete. Choose a selector that represents the data you need. See the rendering options.

When your application needs both content and an image, request a screenshot and page content in the same render. If your destination is a knowledge base or an LLM workflow, our website-to-Markdown tutorial shows how to request a text-oriented output instead.

Handle pagination and lazy-loaded content

The scripts above scrape one page. A full crawl needs an explicit plan for discovering and visiting more URLs.

For pagination, extract the next-page link, resolve it against page.url, and navigate to that URL. Repeat the content readiness check on every page. Set a maximum page count and track visited URLs so a repeated link does not create an endless loop.

For infinite scroll, use bounded scrolling and wait for the item count or a completion marker to change. Stop at a limit even if the page can keep loading. A full-page screenshot captures the available page height; it does not automatically discover every record in an infinite list.

Some applications virtualize their lists and keep only visible rows in the DOM. In that case, extracting the DOM once cannot give you the entire dataset. Collect rows as you scroll, use pagination, or inspect an available data endpoint.

For a Python workflow that also discovers and processes multiple pages, see the Crawl4AI screenshot guide.

Diagnose empty or incomplete results

Keep a screenshot alongside a failed extraction when possible. It often makes a selector problem or an unexpected page state much easier to understand. For image quality and capture settings, continue with the Playwright Python screenshot tutorial.

Frequently Asked Questions

If you read the article, but still have questions. Please, check the most frequently asked. And if you still have questions, feel free reach out at support@screenshotone.com.

How do I scrape a JavaScript website with Python?

Open the page in a browser with Playwright, wait for the elements that contain your data, and extract their text or attributes. You can also request rendered HTML from a service such as ScreenshotOne and parse it in Python.

Can requests and Beautiful Soup execute JavaScript?

No. requests downloads an HTTP response, and Beautiful Soup parses the HTML you give it. For content created by JavaScript, first render the page in a browser or retrieve the underlying data from an available endpoint.

Does waiting for the page load event guarantee that all data is ready?

No. JavaScript can fetch and display data after the load event. Wait for a selector or application state that represents the content you want to extract.