Web scraping with Node.js and Cheerio: static and JavaScript-rendered pages

For web scraping with Node.js, start by fetching the HTML and parsing it with Cheerio. When the page creates its content with JavaScript, render it first and give Cheerio the resulting HTML.

Cheerio provides familiar selectors and methods such as .find(), .text(), and .attr(). It works well for extracting a title, product cards, links, or table rows from a document. It does not run a browser, execute scripts, or create a screenshot.

This guide uses the static and JavaScript versions of Quotes to Scrape. We will reuse one parser with three ways of obtaining HTML: a normal HTTP request, ScreenshotOne, and a local Playwright browser.

Install Cheerio

Use Node.js 22.19 or later, as required by the current Cheerio installation documentation. The examples use Node’s built-in fetch and .mjs files for ES modules.

Create a project and install Cheerio:

1
mkdir cheerio-scraper
2
cd cheerio-scraper
3
npm init -y
4
npm install cheerio

Write a reusable extraction function

Save this as parse-quotes.mjs:

1
import * as cheerio from "cheerio";
2

3
export function parseQuotes(html) {
4
    const $ = cheerio.load(html);
5
    return $(".quote")
6
        .toArray()
7
        .map((element) => {
8
            const card = $(element);
9
            return {
10
                text: card.find(".text").text().trim(),
11
                author: card.find(".author").text().trim(),
12
                tags: card.find(".tag")
13
                    .toArray()
14
                    .map((tag) => $(tag).text().trim()),
15
            };
16
        });
17
}

The parser takes an HTML string and returns plain JavaScript objects. It does not care how the HTML was obtained. Keeping extraction separate from retrieval lets you change the rendering approach without rewriting every selector.

The selectors match this practice site. For another website, inspect its document and substitute the elements that contain your own fields.

Scrape a static page with fetch and Cheerio

Save this as scrape-static.mjs in the same directory:

1
import { writeFile } from "node:fs/promises";
2
import { parseQuotes } from "./parse-quotes.mjs";
3

4
const response = await fetch("https://quotes.toscrape.com/", {
5
    signal: AbortSignal.timeout(30_000),
6
});
7
if (!response.ok) {
8
    throw new Error(`Page returned HTTP ${response.status}`);
9
}
10

11
const html = await response.text();
12
const quotes = parseQuotes(html);
13
if (quotes.length === 0) {
14
    throw new Error("No quotes found; inspect the response and selectors");
15
}
16

17
await writeFile("quotes.json", JSON.stringify(quotes, null, 2), "utf8");
18
console.log(`Saved ${quotes.length} quotes`);

Run it with:

1
node scrape-static.mjs

The static practice page includes its quote cards in the HTML response, so this approach can extract them without launching a browser.

The examples work with UTF-8 HTML. If you are reading pages with other encodings, Cheerio also has loaders for raw bytes and streams. See its document loading documentation.

Why the same scraper can miss JavaScript content

Change the URL in the static script to https://quotes.toscrape.com/js/. That version uses JavaScript to construct its quote cards. The initial document can include scripts and data without containing the .quote elements our parser expects.

Cheerio parses the supplied document. Installing another HTML parser or waiting a few seconds after fetch will not execute the scripts inside that response.

You have several possible paths:

• Retrieve an available data endpoint or embedded JSON when those fields are sufficient.
• Render the page through an API and parse its HTML.
• Use a local browser when you need navigation, clicks, or a custom browser session.

This is a retrieval decision. The parseQuotes function can stay the same once the cards exist in the HTML.

Scrape JavaScript-rendered HTML with ScreenshotOne

ScreenshotOne can render the target page and return its HTML with format=html. Your Node.js application handles the request and Cheerio handles the extraction.

Set your access key from a ScreenshotOne account:

1
export SCREENSHOTONE_ACCESS_KEY="your_access_key"

Save this as scrape-rendered.mjs:

1
import { writeFile } from "node:fs/promises";
2
import { parseQuotes } from "./parse-quotes.mjs";
3

4
const accessKey = process.env.SCREENSHOTONE_ACCESS_KEY;
5
if (!accessKey) {
6
    throw new Error("Set SCREENSHOTONE_ACCESS_KEY before running this script");
7
}
8

9
const requestUrl = new URL("https://api.screenshotone.com/take");
10
requestUrl.search = new URLSearchParams({
11
    url: "https://quotes.toscrape.com/js/",
12
    format: "html",
13
    wait_for_selector: ".quote .text",
14
}).toString();
15

16
const response = await fetch(requestUrl, {
17
    headers: { "X-Access-Key": accessKey },
18
    signal: AbortSignal.timeout(120_000),
19
});
20
if (!response.ok) {
21
    throw new Error(`Rendering request returned HTTP ${response.status}`);
22
}
23

24
const html = await response.text();
25
const quotes = parseQuotes(html);
26
await writeFile("rendered.html", html, "utf8");
27
if (quotes.length === 0) {
28
    throw new Error("No quotes found; inspect rendered.html and the selectors");
29
}
30
await writeFile("quotes.json", JSON.stringify(quotes, null, 2), "utf8");
31
console.log(`Saved ${quotes.length} quotes from rendered HTML`);

Run it with node scrape-rendered.mjs. Keep the access key in the server-side environment.

The wait_for_selector option waits for a matching element to appear in the DOM. For an application that inserts data gradually, use a completion marker that represents the finished content. A first card appearing does not establish that every card has loaded. See the selector wait options.

If you also need an image of the page, the API can capture a screenshot and return content in the same request. Choose HTML metadata for your Cheerio parser, or Markdown metadata for a reading workflow.

Render locally with Playwright, then parse with Cheerio

A local browser is useful when you need to click a button, log in through a custom flow, or inspect the page while developing your scraper.

Install Playwright and Chromium:

1
npm install playwright
2
npx playwright install chromium

Save this as scrape-browser.mjs:

1
import { writeFile } from "node:fs/promises";
2
import { chromium } from "playwright";
3
import { parseQuotes } from "./parse-quotes.mjs";
4

5
const browser = await chromium.launch();
6
try {
7
    const page = await browser.newPage({
8
        viewport: { width: 1280, height: 800 },
9
    });
10
    const response = await page.goto("https://quotes.toscrape.com/js/", {
11
        waitUntil: "domcontentloaded",
12
        timeout: 30_000,
13
    });
14
    if (response && response.status() >= 400) {
15
        throw new Error(`Page returned HTTP ${response.status()}`);
16
    }
17
    await page.locator(".quote .text").first().waitFor({
18
        state: "visible",
19
        timeout: 20_000,
20
    });
21

22
    const html = await page.content();
23
    const quotes = parseQuotes(html);
24
    if (quotes.length === 0) {
25
        throw new Error("No quotes found; check the page and selectors");
26
    }
27
    await writeFile("quotes.json", JSON.stringify(quotes, null, 2), "utf8");
28
    await writeFile("rendered.html", html, "utf8");
29
    await page.screenshot({ path: "quotes.png", fullPage: true });
30
    console.log(`Saved ${quotes.length} quotes and a screenshot`);
31
} finally {
32
    await browser.close();
33
}

Run it with node scrape-browser.mjs. This produces the same structured fields as the other scripts, plus a screenshot for checking the rendered state.

Playwright supplies the browser and Cheerio supplies the parser. You could also extract directly with Playwright locators; Cheerio is useful when you already have parsing functions you want to reuse. Our Playwright waiting guide explains how to pick stronger readiness conditions.

Follow links and process multiple pages

For pagination, look for the next-page link in the HTML after each extraction. Resolve relative links against the page URL with new URL(href, pageUrl). Keep a set of visited URLs and a maximum page count.

Do not assume that all pages have the same loading behavior. A static category page can link to JavaScript-heavy detail pages. You can use ordinary requests for the first group and render only the pages that need it.

When increasing throughput, bound the number of concurrent requests, give each one a timeout, and record failures with their target URL. Start with a small sample before launching a large crawl. ScreenshotOne’s concurrency documentation explains the API-side limit.

Choose HTML, Markdown, or screenshots for the result

For Markdown output, continue with converting a website to Markdown with JavaScript or Python. For screenshots without extraction, use the existing Node.js website screenshot guide.

Frequently Asked Questions

If you read the article, but still have questions. Please, check the most frequently asked. And if you still have questions, feel free reach out at support@screenshotone.com.

Can Cheerio scrape a website that uses JavaScript?

Cheerio can parse HTML produced by a JavaScript application, but it cannot execute the page's scripts. First render the page with a browser or rendering API, then pass the resulting HTML to Cheerio.

What is the difference between Cheerio and Playwright?

Cheerio parses HTML and provides methods for selecting and extracting elements. Playwright controls a browser that can load pages, execute JavaScript, interact with elements, and take screenshots. They can be used together.

Do I need a browser for every Node.js scraper?

No. If the data is in the HTTP response, fetch the HTML and parse it with Cheerio. Use rendering when the fields you need are created by client-side JavaScript or depend on browser interaction.