How Scrape HTML Tabular data with Python

How Scrape HTML Tabular data with Python

Many of you might have already read several articles about data scraping from the websites. Most of them suggested using Node.js with Cheerio library or Python with Beautiful Soup. Although it is very effective when you master the techniques, it takes your time and effort until you finish all the coding for finding an element you need, requesting data, cleaning data to create a dataframe before you can do the actual data analysis. (And, of course, some additional time to fix all the bugs and errors.

This short article will show you a tutorial on how to the easiest way to scrape the tabular data from any website with the three lines of Python Script!

Example of Scraping Real-time COVID-19 Data from Worldometer:


For example, you want to get the tabular data from the Worldometer website. As this dataset is dynamic, changing over time, the Data scraping is make-sense that we get the most updated result every time when running the script!

Example Covid-19 Tabular Data from Worldometer Example Covid-19 Tabular Data from Worldometer

To scrape this dataset, get your machine ready with Python and Pandas. We gonna use the Pandas read_html() to extract all tables of any webpage. However, we cannot just use it to read URL directly because you might face an error 403: Forbidden. To avoid the error, we gonna request it with requests module first to get the HTML body before use Pandas to read it. Overall, the script looks like this:

import requests, pandas as pd
r = requests.get('http://www.worldometers.info/coronavirus/')
dfs = pd.read_html(r.text)

pandas.read_html() function searches for HTML <table> related tags on the input (URL) you provide. It always returns a list, even if the site only has one table.

dfs[0]
         #           Country,Other  TotalCases NewCases  TotalDeaths NewDeaths  TotalRecovered  ActiveCases  Serious,Critical  Tot Cases/1M pop  Deaths/1M pop  TotalTests  Tests/ 1M pop    Population
0      NaN                   World     5622939  +38,672     348715.0    +1,102       2393539.0    2880685.0           53131.0             721.0           44.7         NaN            NaN           NaN
1      1.0                     USA     1709243   +3,017      99883.0       +78        465668.0    1143692.0           17116.0            5167.0          302.0  15204572.0        45961.0  3.308117e+08
2      2.0                  Brazil      376669      NaN      23522.0       NaN        153833.0     199314.0            8318.0            1773.0          111.0    735224.0         3461.0  2.124098e+08
3      3.0                  Russia      362342   +8,915       3807.0      +174        131129.0     227406.0            2300.0            2483.0           26.0   9160590.0        62775.0  1.459285e+08
4      4.0                   Spain      282480      NaN      26837.0       NaN        196958.0      58685.0             854.0            6042.0          574.0   3556567.0        76071.0  4.675305e+07
...
dfs[1]
         #           Country,Other  TotalCases NewCases  TotalDeaths NewDeaths  TotalRecovered  ActiveCases  Serious,Critical  Tot Cases/1M pop  Deaths/1M pop  TotalTests  Tests/ 1M pop    Population
0      NaN                   World     5584267  +90,184     347613.0    +3,096       2362984.0    2873670.0           53167.0             716.0           44.6         NaN            NaN           NaN
1      1.0                   China       82985      +11       4634.0       NaN         78268.0         83.0               7.0              58.0            3.0         NaN            NaN  1.439324e+09
2      2.0                     USA     1706226  +19,790      99805.0      +505        464670.0    1141751.0           17114.0            5158.0          302.0  15187647.0        45910.0  3.308117e+08
3      3.0                  Brazil      376669  +13,051      23522.0      +806        153833.0     199314.0            8318.0            1773.0          111.0    735224.0         3461.0  2.124098e+08
4      4.0                  Russia      353427   +8,946       3633.0       +92        118798.0     230996.0            2300.0            2422.0           25.0   8945384.0        61300.0  1.459285e+08
...