useful SEO XPath's & CSS selectors X
⚠️ THIS IS A WORK IN PROGRESS
If you are crawling website for SEO purposes, whether it is with rcrawler, rvest or another way. You will probably need to use some CSS/Xpath selector to extract the useful bit from each page.
here is a collection of the most useful ones. Feel free to reach out if you think it misses important ones.
Value | xpath | CSS |
Robots (Index/Noindex) | //meta[@name='robots']/@content | head > meta[rel="robots"] |
canonical tag | //link[@rel='canonical']/@href | head > link[rel="canonical"] |
Page Title | //title | head > title |
Meta Description | //meta[@name='description']/@content | head > meta[name="description"] |
Last modified 9mo ago