Puppeteer runs headless by default but can also be configured to run full (non-headless) Chrome or Chromium.

Puppeteer provides you with an API that allows you to take remote control of the headless Chromium instances and use them as a launching point for leveraging the ability of a browser like Chrome to render Javascript elements on a webpage. Like we said above, headless means you are interacting with a CLI rather than a GUI. It’s maintained by the Chrome DevTools team and an awesome open-source community. “What is Puppeteer?” Puppeteer is a Node.js library that provides a high-level API to control headless Chrome or Chromium or to interact with the DevTools protocol. We’ll answer the question “what is Puppeteer?” and how can you use it for web scraping. However, for this article, we’ll be focusing on Google’s Puppeteer for Chrome. There are several examples of headless browser automation tools like Selenium for Firefox, Zombie.js, and Intoli’s Remote Browser. You don’t interact with visual elements on a screen but instead, use a Command-Line Interface to make requests and automate use-cases. It is referred to as headless because there is no Graphical User Interface to interact with. Headless browser automation is a way of leveraging the ability of your browser to render Javascript code for automating use-cases, like web scraping. This is where headless browser automation and Puppeteer come in.

However, seeing as most web scraping tools are designed to capture HTML code and extract data from that, you run into the problem of how to extract website data that is rendered by Javascript code. Essentially, Javascript has become the language of modern websites. When your browser makes a request, the server usually brings a response of Javascript files injected into the HTML framework. Now, this might be good news for the average internet user, but for someone who is looking to perform tasks like automated web scraping, it’s a tad inconvenient. Over the past decade, the internet has evolved from bare-bones websites built with ordinary HTML and CSS to having complex web apps with interactive user interfaces, built using frameworks like Angular or React, that is written with Javascript.

Free download puppeteer github