Requests-HTML

Requests-HTML is a Python library for HTML parsing and web scraping that builds on the Requests HTTP client, adding JavaScript rendering, CSS and XPath selectors, and async support to make scraping "for Humans".

What is Requests-HTML?

Requests-HTML is a Python library that takes an HTTP response or a raw HTML string and returns an HTML object you can query with CSS selectors, XPath, and text search. It runs on Python 3.6 and above, is maintained by the PSF on GitHub, and is installed with pipenv install requests-html. Unlike plain Requests, it can execute JavaScript on a page via Chromium and return the fully rendered DOM.

Key Features

• Full JavaScript support — The render() method uses Chromium (through pyppeteer) to execute JavaScript and update the page HTML, allowing access to dynamically loaded content.
• CSS Selectors — Powered by PyQuery, elements can be selected with jQuery-style syntax like r.html.find('#about', first=True).
• XPath Selectors — The xpath() method supports XPath queries for elements, e.g. r.html.xpath('/html/body/div[1]/a').
• Async support — AsyncHTMLSession enables concurrent requests via asession.run(), with arender() for async JavaScript rendering.
• Link extraction — .links returns all relative links and .absolute_links returns absolute URLs, excluding anchors.
• Text search — The search() method uses a placeholder pattern to extract text snippets from the rendered HTML.
• Connection pooling and cookie persistence — Inherited from the underlying Requests library.
• Mocked user-agent — Requests out of the box sends a browser-like User-Agent string.

Who is it for?

• Data analysts and researchers: extract data from static and JavaScript-heavy pages without needing a full browser automation setup.
• Web developers: validate CSS selector and XPath expressions against live sites during development.
• Python automation engineers: build scrapers that need to handle cookies, redirects, and concurrent fetch workflows.

What can you do with Requests-HTML?

• Scrape dynamic pages: fetch a page, call render(), and then use CSS selectors to pull values from content that only appears after JavaScript runs.
• Concurrent fetching: use AsyncHTMLSession to request multiple URLs at once and process each response as it completes.
• Parse raw HTML offline: instantiate HTML(html=doc) directly from a string to inspect links and elements without making a network request.
• Inventory page links: quickly retrieve a full set of relative and absolute links from any page for crawling or link analysis.

How does Requests-HTML work?

Start by creating an HTMLSession and issuing a GET request. The response's .html attribute gives you a rich object with .find(), .xpath(), .links, and .search(). For JavaScript-rendered content, call .render(), which downloads Chromium into your home directory (e.g. ~/.pyppeteer/) on first run, then executes the page scripts and updates the HTML.

FAQ

Is Requests-HTML free?

Yes, it is an open-source library available through pip, and the project is hosted under the Python Software Foundation on GitHub.

What Python versions are supported?

Only Python 3.6 and above is supported, as stated in the installation instructions.

How do I get the absolute links from a page?

Use the .absolute_links property on the HTML object, which resolves relative URLs to full URLs. The .links property returns them in their original relative or absolute form.

Does Requests-HTML support async JavaScript rendering?

Yes, AsyncHTMLSession provides an arender() method, and all the same parsing operations work on the returned responses.