To prevent Common Crawl from crawling your website, include the following in your robots.txt:User-agent: CCBot
Disallow: /
To prevent Common Crawl from crawling your website, include the following in your robots.txt:User-agent: CCBot
Disallow: /