In this part of the lab we are going to learn how to download and read a text file, as well as a few ways to analyse and extract interesting information about it.
As always we need to open a new R script in which you will save your code for the analysis. When you have opened and saved your scriped on your computer, go ahead and set your working directory:
Session > Set Working Directory > To Source File Location
Next, let us start by downloading the packages we will need in order to do obtain the functions needed for this part of the data retrieval.As we have previously stated, its regarded as “best practice” to have your packages stored in the beginning of the R-script. Some of you might have downloaded these packages before. If you have, you don’t need to run the “install.packages()” function again, but you will need to run the corresponding “library()” function in order to load the package.
install.packages("tidyverse")
install.packages("gutenbergr")
install.packages("tidytext")
install.packages("tm")
install.packages("wordcloud")
install.packages("RColorBrewer")
install.packages("SnowballC")
install.packages("topicmodels")
library(tidyverse)
library(gutenbergr)
library(tidytext)
library(tm)
library(wordcloud)
library(RColorBrewer)
library(SnowballC)
library(topicmodels)
# Note: if you struggle installing packages, try to specify the repoitory. For example:
install.packages("tidyverse", repos='http://cran.rstudio.com/') There are many potential sources of text data that one can use for text analysis. The data we are going to be working with comes from Project Guthenberg, and it is a library that contains more than 7000 free eBooks. These books are free to work with as they no longer have copyright.
We will start by looking at two different techniques on how to download the same type of data.
# Packages used in this step: tidyverse We will start by trying to download the book Dracula by Bram Stoker
# Using the download.file() function we can download a file from the internet:
download.file("https://www.gutenberg.org/files/345/345-0.txt","./file.txt")
# And store it as an object in the global environment:
book <- read_file("./file.txt")The book should now appear in your global environment. We can start by just looking at the file.
head(book)
For the second technique we will use the gutenbergr package to retrieve the data. For this example we are going to try to work with the book Romeo and Juliet.
# Packages used in this step: gutenbergr
# Download using the gutenberg_works function
romeo <- gutenberg_works(title == 'Romeo and Juliet') %>%
gutenberg_download(meta_fields = 'title')
# We can see some of the content of the first lines of the book
head(romeo)We can see that this book is stored in a dataframe (tibble), where we see one of the columns containing the books ID number from the Gutenberg website, one containing the text/content of the book (inlcluding blank spaces), and one telling us the title of the book.
Next, we will do something called “tokenization” of the text, meaning that we will break down the text in to separate words. This can be useful for different types of analyses, such as analyzing the sequence of the words. An example of tokenization would be to change the string “This is an R course” into “This” “is” “an” “R” “course”.
# Packages used in this step: tidytext
# Tokenization:
words <- romeo %>% unnest_tokens(word,text) # we send the dataframe "romeo" into a pipeline %>% to the function unnest_tokens
head(words)Side note: If you are not comfortable with using piping yet (%>%) check out this video.
As we can see, each line of the data set does now consist of one single word, rather than strings of words. So, the new words-tibble (dataframe) contains three columns: One for the id of the book, other for the title of the book and a last one for each word(token) in said book
Before moving on to analysing our text we will change the structure of the dataset in to a corpus (which basically is a collection of documents, but which enables us to make certain analyses more easily).
# Packages used in this step: tm
# Corpus:
romeo_corpus <- Corpus(VectorSource(words))As before, the “romeo_corpus” object should appear in your global environment.
Now we will attempt to do some analysis on this tokenism corpus. The first analysis is to generate a wordcloud that will show us the most frequently used words appearing in the book “Romeo and Juliet”.
Do you think that this will yield any interesting results at this point? Why/why not?
# Packages used in this step: wordcloud = a word-cloud generator; RColorBrewer = for color palettes
# Preparations for the word-cloud
# 1. Turn romeo_corpus into a mathematical matrix describing the frequency of terms occurring in it:
dtm <- TermDocumentMatrix(romeo_corpus) # transforms our word collection into a list, describing the frequency of terms
m <- as.matrix(dtm) # turns it into a matrix
# 2. Inspect m
print(m)
# 3. Sort the terms in m in decreasing order:
v <- sort(rowSums(m),decreasing=TRUE)
print(v)
# 4. Convert v into a dataframe
d <- data.frame(word = names(v),freq=v)
# Have a look at d by clicking on the object in the global environement
# Wordcloud
set.seed(1234) # create reproducible results when writing code that involves creating objects that take on random values
wordcloud(words = d$word, # use the column "words" stored in object "d" as the words in the wordcloud
freq = d$freq, # use the frequency count from the object "d" as the count for word frequency
min.freq = 1, # only include words that occur at least one time
max.words=200, # only inlcude a maximum of 200 words in total in the wordcloud
random.order=FALSE, # plot words according to their frequency (i.e. not random)
rot.per=0.35, # percentage of words rotated 90 degrees in the wordcloud
colors=brewer.pal(8, "Dark2")) # specify the colors we want to use (as stored in RColorBrewer)# Remember that by using the dollar sign we tell r that we want it to use the variable stored in a data frame (one of its columns), for example, we want to access the column 'word' in the dataframe called 'd': d$word Your generated wordcloud should appear in the bottom-right panel of your R-studio under “Plots”. This wordcloud tells us which words are the most commonly used in the book. However, as we can see, we are not necessarily gaining the most interesting information from this, as we see words like “and” and the number 1513 (which is the books eBook number on the Gutenberg webpage) are very common. The reason for this is that we have not done any data cleaning. In this case, it has to do with the removal of excess symbols and/or numbers as well as stop-words (i.e. words that are extremely common in written language but that lack any information value, such as “and”, “but”, “the” etc.). We are therefore going to remove these in this next step.
# Ignore the warning messages
# Convert the text to lower case in order to avoid doubles
romeo_stem <- tm_map(romeo_corpus, content_transformer(tolower))
# Remove numbers
romeo_stem <- tm_map(romeo_stem, removeNumbers)
# Remove common stopwords
romeo_stem <- tm_map(romeo_stem, removeWords, stopwords("english"))
# Remove punctuation
romeo_stem <- tm_map(romeo_stem, removePunctuation)
# Eliminate extra white spaces
romeo_stem <- tm_map(romeo_stem, stripWhitespace)Now, we will attempt to generate the wordcloud anew and see if there has been any changes to the output.
# Note that we are overriding the objects we created earlier.
dtm <- TermDocumentMatrix(romeo_stem)
m <- as.matrix(dtm)
v <- sort(rowSums(m),decreasing=TRUE)
d <- data.frame(word = names(v),freq=v)
# Wordcloud 2
set.seed(1234)
wordcloud(words = d$word, freq = d$freq, min.freq = 1,
max.words=200, random.order=FALSE, rot.per=0.35,
colors=brewer.pal(8, "Dark2"))As we should see in the new wordcloud, the word “and” and the number “1513” have now disappeared from the cloud as a result of our data cleaning. However, we might think that even the words “Romeo” and “Juliet” lack analytical value, as it is obvious that they would be very common in a book with the same title. Because of that, we might want to remove specifically these two words in order to see which other words would then be the most common ones.
romeo_stem2 <- tm_map(romeo_stem, removeWords, c("juliet","romeo")) #remove the words in the vector c( )
# Wordcloud 3
# Note that we are overriding the objects we created earlier.
dtm <- TermDocumentMatrix(romeo_stem2)
m <- as.matrix(dtm)
v <- sort(rowSums(m),decreasing=TRUE)
d <- data.frame(word = names(v),freq=v)
set.seed(1234)
wordcloud(words = d$word, freq = d$freq, min.freq = 1,
max.words=200, random.order=FALSE, rot.per=0.35,
colors=brewer.pal(8, "Dark2"))In our third wordcloud we see a lot more going on and a lot more nuance. Some words that would not have appeared as important before the cleaning are now standing out.
Next, we will make a similar analysis for the books by Aristotle. First we want to find out which books by Aristotle are available on the Gutenberg website.
gutenberg_works(author == "Aristotle")As we can see by the tibble in the output, there are seven books available. In the first column we see the ID-number that the website uses for the book, and in the second column we see the title of the book. In this exercise we are going to work with several books simultaneously. Therefore we will write a code that allows us to download several of these books in to one single vector using the c() function. We are going to specify which books we want to download by specifying their Gutenberg ID.
aristotle <- gutenberg_download(c(1974, 2412, 6762, 6763, 8438))
head(aristotle)You should now find a new R object titled “aristotle” in your global environment. And, using the head() function we can see the first few rows of that object. Similarly to before, we are going to tokenize these texts and turn them in to a corpus to make the analysis a little easier. We will also clean the data in the same way as we did with Romeo and Juliet, before making a wordcloud out of this data.
#tokenize
aristotle_words <- aristotle %>% unnest_tokens(word,text)
#make corpus
aristotle_corpus <- Corpus(VectorSource(aristotle_words))
#Data cleaning:
# Convert the text to lower case
aristotle_stem <- tm_map(aristotle_corpus, content_transformer(tolower))
# Remove numbers
aristotle_stem <- tm_map(aristotle_stem, removeNumbers)
# Remove english common stopwords
aristotle_stem <- tm_map(aristotle_stem, removeWords, stopwords("english"))
# Remove punctuations
aristotle_stem <- tm_map(aristotle_stem, removePunctuation)
# Eliminate extra white spaces
aristotle_stem <- tm_map(aristotle_stem, stripWhitespace)
# Make wordcloud
# Note that we overwrite the former objects.
dtm <- TermDocumentMatrix(aristotle_stem)
m <- as.matrix(dtm)
v <- sort(rowSums(m),decreasing=TRUE)
d <- data.frame(word = names(v),freq=v)
set.seed(1234)
wordcloud(words = d$word, freq = d$freq, min.freq = 1,
max.words=200, random.order=FALSE, rot.per=0.35,
colors=brewer.pal(8, "Dark2"))We can also plot this as a barplot, as another way to visualize it.
barplot(height=head(d,10)$freq, names.arg=head(d,10)$word, # plot the frequency of the top 10 words in the data frame "d" using the function head selecting the head of the df + label the bars accordingly with the top 10 words using the names.arg argument
space = 0.4, # space between the bars, edit this number if your plot is too narrow to see it correctly
xlab="words", # label x-axis
ylab="apparitions", # label y-axis
col="#973232", #pick a colour
main="Aristotle") # title of the graph
Plotting the frequencies of words in that way can be useful in many types of analysis. However, it is not always the case that the most frequent words in a text are the ones that are the most interesting (even after cleaning the data). TF-IDF is a method that builds on word frequencies, but specifically tries to identify the most distinctively frequent and significant words. The logic of TF-IDF is that the words that contain the greatest meaning in a particular document are the words that appear many times in that specific document, but more rarely in others. This means that we are trying to identify the relatively important, but not necessarily common words in a text, compared to other texts.
TF-IDF stands for “term-frequency * inverse-document-frequency” TF thus means the number of times a given term appears in texts IDF stands for the log of the total number of texts / the number of texts with that term.
Lets try this on the books by Aristotle, but lets first install the proper packages, prepare (clean) the data, as well as get an overview of the number of times different words appear in each book.
# Packages used in this step: SnowballC
aristotle_books <- aristotle %>% unnest_tokens(word,text, token = "regex", pattern="\\s+|[[:punct:]]+") %>% #already remove punctuation
mutate(word = str_to_lower(word)) %>% # change to lower case
mutate(stem = wordStem(word)) %>% # stems words
filter(!grepl('[0-9]', word)) %>% # remove numbers
add_count(gutenberg_id,name = "total_words") %>% # add a variable that counts the ttal words in each of the books
group_by(gutenberg_id, total_words) %>%
count(word, sort = TRUE) %>% # sorts by word count (most common word on top)
ungroup()
head(aristotle_books)What we see here using the head() function is the structure of the data where each row represents one word from one of the books. As always, you can also click on the object in the global environment to inspect your output table further. The first column tells us which book it is (by the ID from the Gutenberg webpage), the second column tells us the total number of words in the text, the third column shows the actual word, and the fourth column tells us the number of times that word occurs in this specific text. So far, we see that the “usual suspects” are in the top, such as “the”, “of”, “and” and “is”.
We needed to make these changes to our dataframe so that we can apply the function bind_tf_idf() to it. That function (from the tidytext package) will run the TF_IDF analysis for us. The bind_tf_idf() function is used on “tidy text datasets” i.e. a dataset with one token (word) per row. To run the function, the dataset further needs to have a variable specifying the different source documents and a variable that counts how many times each document contains this token (word). As we have seen above, that is the case for our dataset “aristotle_books” now.
(Worth noting is that, although it is not necessary to remove stopwords when conducting a TF-IDF analysis, as stopwords will generally be common enough in all documents to not appear important anyway (low IDF), it is good practice to do so. However, there can be instances where stop words can have particular meaning in themselves, e.g. the word “her” in an abortion debate. In these cases, the TF-IDF analysis might not be the best option.)
aristotle_books <- aristotle_books %>% # we make changes to our df aristotle_books by sending it into a pipe
select(-total_words) %>%
bind_tf_idf(term = word, document = gutenberg_id, n = n) #
# If you want to learn more about the bind_tf_idf function call
?bind_tf_idf
head(aristotle_books)As we can see using the head() command, the common stopwords words in the top have extremely low values.When inspecting your df (click on the name in the global env.) you can rearrange the order by clicking on the little arrow buttons. Now, you can see the words with higher values.
Let us plot the TF-IDF as graphs in order to visualize which words have the highers TF-IDF values, and thus are used particularly often in this specific work by Aristotle (compared to the other works in our analysis).
# The following function is meant to organise our graphs in a neat way. You don't need to be able to write such a function yourself, you can just copy it and use it in your own graph creation. Alternatively one could create the different graphs separately and organise them in a joint visual with labels afterwards. What the function does roughly: facet bars: this part of the code creates a function that will facilitate the creation of a number of panels and that wraps them together, this can be useful if you want to be able to compare plots and arrange them space efficiently.
# Function for a well-organised graph
facet_bar <- function(df, y, x, by, nrow = 2, ncol = 2, scales = "free") {
mapping <- aes(y = reorder_within({{ y }}, {{ x }}, {{ by }}),
x = {{ x }},
fill = {{ by }})
facet <- facet_wrap(vars({{ by }}),
nrow = nrow,
ncol = ncol,
scales = scales)
ggplot(df, mapping = mapping) +
geom_col(show.legend = FALSE) +
scale_y_reordered() +
facet +
ylab("")
}
#Plot the facet bar
aristotle_books %>% # send the df into a pipe
group_by(gutenberg_id) %>% # group by the data for each book
top_n(15) %>%# select the top 15 words
ungroup() %>% # ungroup
facet_bar(y = word, # use the graphing function created earlier, specify the y-axis of your graphs = the word
x = tf_idf, # specify the x-axis = the frequency of the word
by = gutenberg_id, # group so that you get one plot per book
nrow = 3) These plots shows us the words with the highest TF-IDF value per book (including stopwords). If we want to do the same analysis without the stop words we do the following:
# Removes the stopwords
aristotle_books_no_sw <- anti_join(aristotle_books,stop_words)
# Plot the facet bars
aristotle_books_no_sw %>%
group_by(gutenberg_id) %>%
top_n(15) %>%
ungroup() %>%
facet_bar(y = word,
x = tf_idf,
by = gutenberg_id,
nrow = 3)As we can see, these plots are very similar to each other, meaning that the stop words have little effect on the results.
In this part we are going to be working with another type of text
material, namely social media data from Twitter. On this material we
will learn how to perform sentiment analysis and latent dirichlet
allocation (LDA) topic modelling.
Lets start by retrieving data from a public dataset of tweets. We will not collect our own Twitter data here since the cleaning of such datasets can take a long time.
tweets <- readRDS(gzcon(url("https://github.com/cjbarrie/CTA-ED/blob/main/data/sentanalysis/newstweets.rds?raw=true")))
head(tweets)In this dataset we find tweets collected from the Twitter accounts of the top eight newspapers in the UK by circulation. These newspapers are:
As we see from using the head() function above, the dataset is quite big and contains a number of different variables. To get a better overview we can use the colnames() function to print out the names of the columns.
colnames(tweets)Each row in the dataset is a tweet produced by one of these newspapers over a five month period (January-May 2020). We will not need all of these variables, so in order to make the dataset less cluttered we will keep only the ones that are useful for this lab.
# We select the variables we want to keep, and then we rename them into variable names that are more intuitive for our purposes
tweets <- tweets %>%
select(user_username, text, user_name) %>% # selecting the 3 variables: user_username, text, user_name
rename(username = user_username, # rename the variables (newname = oldname)
newspaper = user_name,
tweet = text)
head(tweets)As we now see in the output, the dataset only consists of three variables: the name of the newspaper, the username of that newspaper on Twitter and the content of the tweets.
In order to facilitate a sentiment analysis we are going to add a few new variables to the dataset. We are, going to add one variable that we call “word”, which breaks out a word from the tweets in an effort to “tokenize” it, as we have done before. We will also add a variable called “stem”, which is the “word stem” of the word in the “word” column (which makes that different versions of the same word would be connected through the same word-stem, e.g. lose, losing, lost).
tidy_tweets <- tweets %>%
mutate(desc = tolower(tweet)) %>% # change all letters to lower case
unnest_tokens(word, desc, token = "regex", pattern="\\s+|[[:punct:]]+|http.+ |http.+$") %>% # remove punctuation + tokensize
mutate(stem = wordStem(word)) %>% # stems words
filter(str_detect(word, "[a-z]")) # select only tokens (words) starting with a letter a-z, aka get rig of emojis etc.
head(tidy_tweets)Similarly to our previous analyses we will clean the dataset a little by removing stopwords.
tidy_tweets <- anti_join(tidy_tweets,stop_words)## Joining, by = "word"head(tidy_tweets)As we can see by the head() function, the words that are now in the top are a little bit different than they were before, as some stopwords have dropped out.
Next, in order to do sentiment analysis we need to get some sentiment dictionaries. The idea with conducting a sentiment analysis is to classify some material according to the emotions expressed in them. Sentiment dictionaries usually classify words in to different types of emotions. In this analysis we will use a sentiment dictionary called “Bring”, and this particular dictionary classifies words in to the sentiment-categories “positive” or “negative”.
#Get the sentiment lexicon
sentiment <- get_sentiments(lexicon = "bing")
head(sentiment)Here, we see a sneak-peak of some of the words from the sentiment lexicon.
In order to make calculations easier, we will code the category “positive” as 1, and the category “negative” as -1.
# Recode the categories from positive/negative" to 1/-1
sentiment <- sentiment %>%
mutate(valor = if_else(sentiment == "negative", -1, 1))
head(sentiment)We can now see that the “valor” for the sentiment has changed in to a numerical value instead.
By having a tidy (one word per row) dataset, using a “join” function we will add sentiment to each word in our twitter dataset. Technically speaking we are merging the two dataframes by joining the dataframe “sentiment” to the dataframe “tidy_tweets”. All those words for which there is no sentiment-information available will be automatically filtered away.
tweets_sent <- inner_join(x = tidy_tweets, y = sentiment) # merge the two dataframes
head(tweets_sent)As we can see in the output, the sentiment and valor has now been added to our dataframe for each of the words in the word-column.
Lets add up the sentiments of the words for each tweet to get the full picture. Thus, we will have one value (-1 or 1) per Tweet (instead of a value per word).
tweets_sent %>% group_by(newspaper, tweet) %>% # group by tweet and newspaper
summarise(sentiment = sum(valor)) %>% # add the values of the tokens in each tweet together
head() # look at the headThe output now shows us the sentiment for the full tweet.
Next, we might want to calculate the percentage of positive, negative and neutral tweets for each newspaper in our dataset.
sentiment_percentage <- tweets_sent %>% group_by(newspaper, tweet) %>%
summarise(sentiment = sum(valor)) %>%
group_by(newspaper) %>%
summarise(pos = 100 * sum(sentiment > 0) / n(),
neutral = 100 * sum(sentiment == 0) / n(),
neg = 100 * sum(sentiment < 0) / n())
head(sentiment_percentage)The output now shows us the percentage of positive (sentiment score >1 ), negative (sentiment score <1) or neutral (sentiment score = 0) tweets for each of the news outlets. Lets try and plot this as a graph.
sentiment_percentage %>% ungroup() %>%
gather(key = "sentiment", value = "valor", -newspaper) %>%
ggplot(aes(x = newspaper, y = valor, fill = sentiment)) +
geom_col(position = "dodge", color = "black") + coord_flip() +
theme_bw()
In this final section, we will perform something called the Latent Dirichlet allocation (LDA) topic modelling. LDA is one of the most common algorithms for topic modeling. Without diving into the math behind the model, we can understand it as being guided by two principles:
Every document is a mixture of topics. We imagine that each document may contain words from several topics in particular proportions. For example, in a two-topic model we could say “Document 1 is 90% topic A and 10% topic B, while Document 2 is 30% topic A and 70% topic B.”
Every topic is a mixture of words. For example, we could imagine a two-topic model of American news, with one topic for “politics” and one for “entertainment.” The most common words in the politics topic might be “president”, “congress”, and “government”, while the entertainment topic may be made up of words such as “movies”, “television”, and “actor”. Importantly, words can be shared between topics; a word like “budget” might appear in both equally.
LDA is a mathematical method for estimating both of these at the same time: finding the mixture of words that is associated with each topic, while also determining the mixture of topics that describes each document. There are a number of existing implementations of this algorithm, and we’ll explore one of them in depth.
Before getting started, we need to format our data for the analysis:
tidy_tweets <- tidy_tweets %>% dplyr::count(tweet, word)
dtm_tweets <- tidy_tweets %>% cast_dtm("tweet","word","n")
dtm_tweetsFrom the file summary of ‘dtm’ file we can see that it contains total 167621 documents, which is the total number of tweets that we have, and total 236211 term, which shows we have total 236211 unique words in our tweets.
Next, we need to install a package for topic modelling. We have not included this earlier as this step can cause some problems. Try running the lines one after another. If it does not work, try updating R.
# Installing LDM package
system2('sudo', 'apt-get install libgsl0-dev')
install.packages('topicmodels', repos='http://cran.rstudio.com/')
library(topicmodels)# Note: The calculation will take a while
# LDA model of 5 topics:
lda_5 <- LDA(dtm_tweets, # input object
k = 5, # k specifies the number of topics we want to extract
method = 'Gibbs', # method used for fitting
control = list(seed = 1234)) # set a seed so that the output of the model reproducible
#Top 10 terms or words under each topic
top10terms_5 <- as.matrix(terms(lda_5,10))
top10terms_5
#number of topics found out by our model:
lda.topics_5 <- as.matrix(topics(lda_5))
summary(as.factor(lda.topics_5[,1]))
#We can also get document wise probability of each topic
topicprob_5 <- as.matrix(lda_5@gamma)
head(topicprob_5,1)As a sample we can see that according to my model with 5 topics, how document 1 has different probabilities of containing each topic. The highest probability from topic-1.