Download and check XML sitemaps

It's not required to submit an XML sitemap to have a successful website but it's definitely an SEO nice to have. 
Nevertheless, if you do submit one, it's best to make sure it's error-free and as you will see its is quite straightforward to extract URLs using R
Install xsitemap R’ Package (to be done once) and Load
1
# Installing libraries and Loading libraries
2
install.packages("devtools")
3
library(devtools)
4
install_github("pixgarden/xsitemap")
5
library(xsitemap)
6
​
Copied!
Find and fetch XML sitemaps
1
xsitemap_urls <- xsitemapGet("https://www.nationalarchives.gov.uk/")
Copied!
This function will first search for XML sitemap url. It will first check the robots.txt file to see if an XML sitemap url is explicitly declared.
if not, the script will do some random guess (‘sitemap.xml’, ‘sitemap_index.xml’ , …) most of the time, it will find the XML sitemap url.
Then, the XML sitemap URL is fetched and the URLs extracted.
If it’s a classic XML sitemap, a data frame (special kind of array) will be produced and return.
If it’s an index XML sitemap, the process will get back from the start with every XML sitemaps inside.
This will produce a data frame with all the information extracted. 
Check URLs HTTP code
Another interesting function allows you to crawl the sitemap URLs and verify if your web pages send proper 200 HTTP codes, using HEAD Requests (easier on the website server)
It can take some time depending on the number of URLs. It took several hours for https://www.gov.uk/ for example.
1
xsitemap_urls_http <- xsitemapCheckHTTP(xsitemap_urls)
Copied!
It will add a dedicated column with the HTTP code filled in. You can check data inside rstudio by using 
1
View(xsitemap_urls_http)
Copied!
or if you prefer, generate a CSV 
Count HTTP codes
Like in the intro, it's quite easy to count HTTP codes
1
View(table(xsitemap_urls_http$http))
Copied!
to discover, at the time of writing that most of the XML sitemap URLs are actually redirects...
Plot the years the pages were added
You might have noticed that in this XML sitemap with a "lastmod" field. This is an optional field that explicitly declares to Google last modification date. This allows theoretically Google to optimise website crawls.
It also allows us to understand how fresh is one's website content as we can plot it
1

library(ggplot2)
2
​
3


4
ggplot(xsitemap_urls) +
5
 aes(x = lastmod) +
6
 geom_histogram(bins = 90L, fill = "#112446") +
7
 theme_minimal()
Copied!
(I've got help from the esquisse library)
Most of the content originated from 2014-2015, The oldest page have been updated in 2001
Let's try to get a clearer picture by extracting years
1
# We extract from the Date the year and
2
# store the value into a new column called 'year'
3
xsitemap_urls$year <- format(xsitemap_urls$lastmod,"%Y")
4
​
5
​
6
# Removing not available values (NA's)
7
# and ploting the url by year
8
xsitemap_urls %>%
9
 filter(!is.na(year)) %>%
10
 
 ggplot() +
11
  aes(x = year) +
12
  geom_bar(fill = "#112446") +
13
  theme_minimal()
14
  
Copied!
If you prefer a % cumulative view:
1
plot(ecdf(xsitemap_urls$year))
Copied!

Crawl and extract data - Previous

What's crawling and why is it useful?

Next - Crawl and extract data

Crawling with rvest

Last modified 3mo ago

Copy link

Contents

Install xsitemap R’ Package (to be done once) and Load

Find and fetch XML sitemaps

Check URLs HTTP code

Count HTTP codes

Plot the years the pages were added