USE THE ROBOTS.TXT FILE TO MANAGE
YOUR CRAWL BUDGET
Crawl budget optimization is an important part of making sure that
Google can crawl your site efficiently and easily.
You are making it more efficient and easier for Google to crawl your
site through this process. You optimize your crawl budget in two
ways – the links on your site and robots.txt.
The method that primarily concerns us at this step is using robots.
txt for crawl budget optimization.
This guide from Google tells you everything you need to know
about the robots.txt file and how it can impact crawling.
GOOGLE WEBMASTER GUIDELINES: EVERYTHING YOU NEED TO
KNOW & UNDERSTAND