Scrape Table Data from the Web

Posted by Ben Starr on Jul 13, 2017 2:57:00 PM
Ben Starr
Find me on:

There is a wealth of data available on the web that dynamically updates, but is not published as a neat .csv file that works with eagle.io out of the box.

An example of this type of data is river heights published by BOM in near real-time in a HTLM table http://www.bom.gov.au/fwo/IDQ65388/IDQ65388.540319.tbl.shtml

This article shows you how to import this data to eagle.io using Google Sheets as middle ware.

River height monitoring - HTML Table that updates up to every 2 minutes

Step 1 — Import data to google sheets using IMPORTHTML function.

IMPORTHTML(url, query, index)

  • url - The URL of the page to examine, including protocol (e.g. http://).
  • The value for url must either be enclosed in quotation marks or be a reference to a cell containing the appropriate text.
  • query - Either "list" or "table" depending on what type of structure contains the desired data. Also enclosed in quotation marks.
  • index - The index, starting at 1, which identifies which table or list or table to be returned.

Step 2 — We’ll configure our Spreadsheet to publish the data; this allowseagle.ioto access the information collected by our Form.

Step 3- Choose the specific sheet with data (default is “Entire Document”) and CSV as the format and copy the URL to the published data.

Topics: Environmental data formats, Environmental data mining

    Connect, configure and control your data loggers from the cloud

    Eagle.io is an all-in-one cloud software for hardware integrators involved in remote environmental monitoring. We believe it is critical to monitor natural assets smartly to protect and manage them effectively.

    • Bulletproof IoT networks. Relevant data anytime, from any hardware, to anyone, on any device.

    • Scalable cloud database collection, storage, analysis, mapping, alarming, editing and visualisation of time series sensor data.

    • Trusted by AECOM, CSIRO, Qld Department of Natural Resources, BMT Global, Ecotech, Xylem, Geosyntec, Specto.

    • Easy integration with standard industry loggers.

    • Simple management of discrete networks: White-labeling, managed accounts, in-app customer support, intuitive data configuration and scalable pricing.

    To learn more, register for a 30-day trial and personalised walkthrough at https://eagle.io/trial

    Subscribe Here!

    Resent Posts