useful SEO XPath's & CSS selectors X

⚠️ THIS IS A WORK IN PROGRESS

If you are crawling website for SEO purposes, whether it is with rcrawler, rvest or another way. You will probably need to use some CSS/Xpath selector to extract the useful bit from each page. here is a collection of the most useful ones. Feel free to reach out if you think it misses important ones.

Value	xpath	CSS
Robots (Index/Noindex)	//meta[@name='robots']/@content	head > meta[rel="robots"]
canonical tag	//link[@rel='canonical']/@href	head > link[rel="canonical"]
Page Title	//title	head > title
Meta Description	//meta[@name='description']/@content	head > meta[name="description"]