Scrape Web Pages with YQL and Apps Script

Some web services, Google Search and Amazon Prices for example, may not offer APIs or, if they do, not every detail available on the website pages may be available through the API. In such cases, you can use web scraping with YQL (Yahoo Query Language) and Google Scripts to extract any data from their web pages.

You need to specify the URL of the page that you wish to scrape and also the XPath of the element that should be extracted. If you are not familiar with XPath, use the Chrome Dev Tools to inspect the element, right click the node in the DOM tree and choose Copy XPath to know the XPath (see screenshot).

scrape-web-pages

In the snippet below, we are fetching the home page of the New York Times technology section as a JSON though YQL and the results are parsed with Google Apps Scripts.