You may not want sure internet pages of your internet site crawled mainly because they might not be helpful to consumers if located in a internet search engine's search engine results. Be aware that if your site uses subdomains and you want to possess sure webpages not crawled on a selected subdomain, you will have to make a separate robots.Google