You run into a site that has the following line in its robots.txt file:
Disallow: /
This means that the robots.txt file is disallowing crawling from the
top of the site down.
You run into a site that doesn’t have a sitemap.xml directive in
robots.txt. This is considered an SEO best practice.
Delete that line.
Make sure you add in a directive declaring the location of your site-
map file, such as the following:
Sitemap: https://www.example.com/sitemap.xml
Common Issue
Common Issue
Solution
Solution
GOOGLE WEBMASTER GUIDELINES: EVERYTHING YOU NEED TO
KNOW & UNDERSTAND