useful SEO XPath's & CSS selectors X

⚠️ THIS IS A WORK IN PROGRESS

If you are crawling website for SEO purposes, whether it is with rcrawler,  rvest or another way. You will probably need to use some CSS/Xpath selector to extract the useful bit from each page.

here is a collection of the most useful ones. Feel free to reach out if you think it misses important ones.
Value
xpath
CSS
Robots (Index/Noindex)
//meta[@name='robots']/@content
head > meta[rel="robots"]
canonical tag
//link[@rel='canonical']/@href
head > link[rel="canonical"]
Page Title
//title
head > title
Meta Description
//meta[@name='description']/@content
head > meta[name="description"]

Resources - Previous

Execute R code online

Last modified 5mo ago

Value	xpath	CSS
Robots (Index/Noindex)	//meta[@name='robots']/@content	head > meta[rel="robots"]
canonical tag	//link[@rel='canonical']/@href	head > link[rel="canonical"]
Page Title	//title	head > title
Meta Description	//meta[@name='description']/@content	head > meta[name="description"]