Block AI CrawlersThe plugin tells artificial intelligence crawlers (such as OpenAI ChatGPT) not to crawl your website content for training AI. The specific method is to update the websiterobots.txt, to block common AI crawlers. Artificial intelligence crawler will read the websiterobots.txtto check if there are any requests that are not indexed.
It blocks these AI crawlers and bots:
- ChatGPT and GPTBot– Crawlers and web browsers used by OpenAI
- Google Extended– A crawler used for artificial intelligence training on Google Gemini (formerly Google Bard)
- FacebookBot– Crawler for Facebook AI training
- CommonCrawl– Compile crawlers for datasets used to train AI models
- Anthropic AI/Claude– The crawler used by Anthropic
- Omgili– Omgili crawler for artificial intelligence training
- Bytespider– TikTok crawler for artificial intelligence training
- PerplexityBot– Used by Perplexity in its artificial intelligence products
- Applebot– used by Apple to train its artificial intelligence products
- Cohere– Cohere crawler for artificial intelligence training
- DiffBot– Diffbot crawler for artificial intelligence training
- Imagesift– Imagesift crawler for images
Experimental meta tags
The plugin also adds the “noai, noimageai” tag to the site’s meta tags. These tags tell AI bots not to include your content as part of their data set. These are experimental and not yet standardized.
Disclaimer
Notice:Although the plugin adds these tags, compliance with this tag requirement is up to the crawler itself.