2023_CSS_Course_Webscraping

Step 1: Set-up

First, we need to create a new R script and save it in our project folder. If you have not yet created a project folder for this course check out the script of lab 1. Next, set working directory and load necessary packages. We will need the httr package that allows retrieving data from webpages, and tidyverse package to clean the scraped pages from html tags and unnecessary information.

Either click on: Session > Set Working Directory > To Source File Location or use the code: setwd() # include the file path to your working directory in the parentheses

install.packages("httr", repos = "http://cran.us.r-project.org")
install.packages("tidyverse", repos = "http://cran.us.r-project.org")

library(httr) 
library (tidyverse)

## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.2 ──
## ✔ ggplot2 3.4.0      ✔ purrr   1.0.1 
## ✔ tibble  3.1.8      ✔ dplyr   1.0.10
## ✔ tidyr   1.2.1      ✔ stringr 1.5.0 
## ✔ readr   2.1.3      ✔ forcats 0.5.2 
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()

Step 2: Inspecting the material

In this exercise, we want to scrape material on migration politics from the website of the Swedish government. More precisely, we will download pages that include documents about migration politics in Sweden.

Therefore, we need to acquaint ourselves with the target website first and search for the documents manually. We are interested in several specific document types (legal documents, speeches and statements, to name a few).

When you copy the following link into your browser you can see the results for several search key words. Check for: - The number of search results - The number of pages on which the search results are listed - Open some of the results and look at them too

# We will he store the link to the search results stores as an object called "url"
url <- "https://www.government.se/search/?query=migration&ct=Article&ct=Information%20material&ct=Legal%20document&ct=Legal%20document&ct=Press%20release&ct=Speech&ct=Statement"

When looking at the results on the website you might have noticed that the search hits are listed on several pages. At the time of creating this lab script there are four pages storing up to twenty results each: 1 - 20 ; 21 - 40 ; 41 - 60 ; 61 - 73.

Step 3: Extract the search page result URLs

In a next step, we want to generate urls to each of these pages.

# Generating links to all four pages of search results 
urls <- paste0("https://www.government.se/search/?page=", 1:4, "&query=migration&ct=Article&ct=Information%20material&ct=Legal%20document&ct=Legal%20document&ct=Press%20release&ct=Speech&ct=Statement")

# Remember if you want to learn more about a function such as paste you can type help(paste0) into your console

You can now see that we created a new object in the global environment called urls. If you want to see its content check out:

print(urls)

Step 4: Retrieving urls for each article

Each of my four overview pages includes 20 search hits, that are links to 20 result pages. We will call the content of that result page an ‘article’ for now.

In the code below, we loop through each website page, extract links to the articles and save them in a vector.

#create an empty list container for html code
html_code <- vector ("list") 


# Loop through search pages and collect the html code that makes up the website
for (i in 1:length(urls)) { #go through each of the four pages
#access the page to retrieve all kind of information (including meta-data about our request)
  page <- httr:: GET (urls[[i]]) 
#extract content, meaning extracting the html code out of which the website is build
  page_content <- httr::content(page, "text") 
#add raw page content to the list with future article urls
  html_code[[i]] <- page_content 
}

Let’s check what one website page looks like.

#Don't worry, it looks scary! ([[1]] selects the first item in our article_urls list)
html_code[[1]]

Now, we need to get rid of auxiliary information - we only need to extract links to the documents. If we inspect the raw html files (as in the example above), we can find the necessary urls inside the < h3 > tags. So, we need to extract those and remove the rest of html code. We need to use regular expressions (regex) to extract the necessary information. However, learning regex requires some time. In this exercise, we are mostly using a combination (?S).*? inside the html tags of interest - this roughly means “match any number of characters including new line until the first instance of…”. To get some help with regex, you can for example, use the website https://regex101.com/.

Since finding the right html tags is also quite a challenging task, you can install some Chrome extensions, such as SelectorGadget that helps easily identifying the needed html tags - you just need to move your cursor to the website section that you want to extract.

Also, since we are planning to do several manipulations with each link, we are using pipe operator %>% that takes each extracted link and applies several functions in one shot - thus, we don’t need to store multiple interim objects.

# For presentation purposes these libraries had to be loaded again, this is not necessary for you in your programming. 
library(dplyr)
library(magrittr)

## 
## Attaching package: 'magrittr'

## The following object is masked from 'package:purrr':
## 
##     set_names

## The following object is masked from 'package:tidyr':
## 
##     extract

library(knitr)
library(stringr) 
library(httr) 
library (tidyverse)


#create a list that will be filled with future addresses
urls_final <- vector ("list") 


#go through the html-code of each search page each page
for (i in 1:length (html_code)) { 
urls_final [[i]] <- str_extract_all (html_code[[i]],"<h3>(?s).*?</h3>") [[1]] %>% 
#extract the link inside the url html tag to get rid of auxiliary characters
  str_extract_all (., "<a href=(?s).*?</a>") %>% 
# In this step we want to change the object from a list to a flat vector
#delete <a href>
  str_remove_all (., '<a href=\"') %>% 
#delete everything after "
  str_remove_all (., '\\".*')%>% 
#paste "www.government.se" to get complete addresses
  paste0("https://www.government.se", .) 

}

Let’s check the results!

urls_final

Step 5. Scraping the article text

What we see here is that we now have an object containing links to each of the articles. Imagine that we are now going to go click on the headline of an article, enter the article page, and we want to scrape the content of each article. We have thus now left the search result page. In shoet, now, we need to access each link and scrape the text of the corresponding web page. This will take a few minutes.

Broken down in to several steps we need to 1) extract the html code of each article page; 2) in the article html code find try to the section containing the textbody of the document; 3) in order to have a neat dataset we will also want to find the section of the html code that contains the agency that published the document.

First, we want to change the list “urls_final” from a list to a flat vector. In the list, the urls are still stored according to the search pages from which we scraped them. We no longer need this division and rather store all urls in one long vector, and we would like to have the content of urls_final in the form of one long character string. .If you want to understand the unlist() function better check out this video!

# Transform the list into a flat vector 
urls_final <- unlist (urls_final)

# In order to make sure that it is now indeed a vector, we will check it using the is.vector() fucntion

is.vector(urls_final)

## [1] TRUE

We can now check the website of the Swedish government to learn that the body of the document is stored inside the html-tags < div class=\“has-wordExplanation\” > < /div > , while the name of the agency is stored inside the html-tags < div class=\“categories-text\” > < /div > . Lets look at this together at the example of one of the articles using the “inspect” option in your browser and the selector gadget (for those of you using Chrome).

Now that we have located the right html-tags, we can use them in order to continue the webscraping.

First we want to create an empty data frame in which we will store all our scraped information later.

Data frame: Data frames are a specific type of list. This is the format that we usually see our data when we import it to R- an equal number of rows and columns.

# Creating an (almost) empty data frame in which information will be stored later

#first column (variable) stores our article urls
article_df <- data.frame (url = urls_final, 
#variable 2: will store the text of the article, empty for now
                          text = NA,
#variable 3: will store the name of the agency that published the article, empty for now
                          agency = NA,
#variable 4: number of words in the article, empty for now
                          length = NA,
# by default the function data.frame() wants to convert any string data to factorial data. As we want to keep our strings as strings we will tell R manually to not transform string to factors. 
                          stringsAsFactors = F)

To have a look at your data frame, click on the newly created object (article_df) in your global environment.

The following loop will scrape the missing information for the remaining three variables and store them in our dataframe.

# We want to loop through every row in our data frame, since our different urls are stored on different rows, We ask R to print each row as we loop through it in order to be able to follow where we are in the porcess. 
for (i in 1:nrow (article_df)) {
  print(i)
# Now, we will access each article-page to retrieve all kind of information (including meta-data about our request). This will be stored in object "raw_file", and will be stored as raw-bytes. 
       raw_file <- httr:: GET (article_df [i, 1])
# Now, we extract the content-part from the raw_file object, meaning extracting the html code out of which the website is build 
       article_content <- httr::content(raw_file, "text") 
# In this part of the code we...
#extract the text body inside the html-tag we identified before
       article_df [i,2] <- str_extract_all (article_content, '<div class=\\"has-wordExplanation\\">(?s).*?</div>')%>% 
#remove anything inside <> for the text body
         str_remove_all (., "<.*?>") %>% 
#remove any extra whitespaces in the text body
         str_squish () 
#extract all the content from the part of the html code that corresponds to the agency authoring the text
       article_df [i,3] <- str_extract_all (article_content, '<div class=\\"categories-text\\">(?s).*?</a></div>')%>%
##remove two last brackets in the string
         str_remove_all (., "</a></div>") %>% 
#remove anything inside <> for the authoring agency
         str_remove_all (., "<.*>") 
#calculate number of words in the article
       article_df [i,4] <-  str_count(article_df[i,2], '\\w+') 
#ask the system to "fall asleep" for a few seconds, so that the website does not block us.
       Sys.sleep(1+sample(c(0.5,1,1.5,2),1)) 
     }

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : argument is not an atomic vector; coercing

Now we have our data frame with all the necessary information! Let’s check the result:

article_df

Looks like there are some empty cells in the data frame, in particular those that do not contain any article-text. These documents are thus uninformative and we need to get rid of them:

# We tell R to only keep observations that has a length that is greater than zero, thus excluding empty row
df <- article_df[article_df$length >0,]

# As we see in the data set, some observations are not just empty, but has a text string telling us that there is no characters in this observation (character(0)). As this is also not very informative, we want to get rid of these observations as well. 
df <- df[df$text != "character(0)",]

Now, when we are done with some basic cleaning of the data, we can now do some data exploration.

# Lets start by looking at how many different agencies that have been authoring these articles. 
table (df$agency)

## 
##                       character(0)                 Government Offices 
##                                  5                                  7 
##       Ministry for Foreign Affairs Ministry of Climate and Enterprise 
##                                 22                                  1 
##             Ministry of Employment                Ministry of Finance 
##                                  2                                  1 
##                Ministry of Justice            Prime Minister's Office 
##                                  2                                 19 
##                    Ulf Kristersson 
##                                  1

# Finally, lets look at the average document length using some summary statistics.
summary (df$length)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       1     104     378    1040     953    7051

We can also plot this

# Plotting with the base R graphics x = article y = length 
plot(article_df$length)

# Plotting using the ggplot2 package  x = article y = length 
ggplot(article_df, aes(x=url, y=length)) +
  geom_bar(stat = "identity")

# Plotting using the ggplot2 package  x = agency y = length   
ggplot(article_df, aes(x=agency, y=length)) +
  geom_bar(stat = "identity")

# You can skip this step, if you are still struggeling to download ggplot2!

Introduction to Computational Social Science

Webscraping

Step 1: Set-up

Step 2: Inspecting the material

Step 3: Extract the search page result URLs

Step 4: Retrieving urls for each article

Step 5. Scraping the article text