Retaining only the web crawling blocker

If you only want to configure web crawlers using the robots.txt file, and not deploy the Google Site Search engine on your instance, you can request the Remove Google Custom Search update set from ServiceNow support.

This update set removes the Google search functionality but leaves the web crawler configuration capability.