You run into a site that has the following line in its robots.txt file:Disallow: /This means that the robots.txt file is disallowing crawling from the
top of the site down.You run into a site that doesn’t have a sitemap.xml directive in
robots.txt. This is considered an SEO best practice.Delete that line.Make sure you add in a directive declaring the location of your site-
map file, such as the following:Sitemap: https://www.example.com/sitemap.xmlCommon Issue
Common Issue
Solution
Solution
GOOGLE WEBMASTER GUIDELINES: EVERYTHING YOU NEED TO
KNOW & UNDERSTAND