Crawl and extract data
Grabbing data from APIs
Export and read Data
data wrangling & analysis
Send requests to the Google Indexing API using googleAuthR
This guide has been written by Ruben Vezzoli in July 2020. Ruben is a 25 years old , Italian, Data Analyst checkout his website with other interesting R scripts.
Two years ago, Google introduced the Indexing API with the intent of solving an issue that affected jobs/streaming websites – having outdated content in the index. The Google Developers documentation says:
“You can use the Indexing API to tell Google to update or remove pages from the Google index. The requests must specify the location of a web page. You can also get the status of notifications that you have sent to Google. Currently, the Indexing API can only be used to crawl pages with either job posting or livestream structured data.”
Many SEOs are using Indexing API also for non-job-related websites and that’s why I decided to build an R script to try out the API (and it worked).
I’m going to show you how the script works, but don’t forget that there is a free quota of 200 URLs sent per day!
Create the Indexing API Credentials
First of all, you have to generate the client id and client secret keys for the APIs.
Open the Indexing API page and enable the API. Then go to the Credentials page and there you’ll find your credentials.
googleAuthR package & Indexing API options
I created the script using the googleAuthR package, which allows you to send requests to Google APIs.
The code takes in input, a character vector of maximum 200 URLs and returns in a data frame the response of the API (see the screenshot below).
You can use the script to update or delete pages. You just have to change the line 38:
- Use type = “URL_UPDATED” if you have to update the page
- Use type = “URL_DELETED” if you have to remove the page
1
# LAST UPDATE: 20-03-2021
2
# Install & Load the packages
3
4
install.packages("googleAuthR")
5
install.packages("tidyverse")
6
install.packages("readr")
7
8
library("googleAuthR")
9
library("tidyverse")
10
library("readr")
11
12
# Set credentials and scope
13
14
clientId <- "PASTE HERE YOUR CLIENT ID"
15
clientSecret <- "PASTE HERE YOUR CLIENT SECRET"
16
scope <- "https://www.googleapis.com/auth/indexing"
17
18
options("googleAuthR.client_id" = clientId,
19
"googleAuthR.client_secret" = clientSecret,
20
"googleAuthR.scopes.selected" = scope,
21
"googleAuthR.verbose" = 0 # Not mandatory - I just use it to debug the script
22
)
23
24
# Google API OAuth
25
26
gar_auth()
27
28
# List of URLs - Daily Limit of 200 URLs
29
30
urls <- read_csv("~/Desktop/Your-file-name.csv")
31
urls <- urls[ ,1]
32
33
# indexingApi function - you can use the function to send requestes to the indexing API using urls vector as an input
34
# It also GET the response from the API and stores it in a data frame
35
36
indexingApi <- function(page) {
37
38
body <- list(
39
url = page,
40
type = "URL_UPDATED")
41
42
f <- gar_api_generator("https://indexing.googleapis.com/v3/urlNotifications:publish",
43
"POST")
44
45
result <- f(the_body = body)
46
result <- as.data.frame(result[[6]][[1]][[2]])
47
48
return(result)
49
50
}
51
52
# The API responses are stored in a data frame
53
54
APIResponse <- map_dfr(
55
.x = urls,
56
.f = indexingApi)
57
58
# You can download the API responses as .csv file
59
60
write.csv(APIResponse, "Your-file-name.csv")
Copied!