# www.robotstxt.org/ # http://code.google.com/web/controlcrawlindex/ User-agent: AhrefsBot Crawl-Delay: 5 User-agent: * Crawl-delay: 2 Sitemap: https://www.provokemedia.com/sitemap.xml Sitemap: https://www.provokemedia.com/gnewssitemap.xml User-agent: Amazonbot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: * Allow: / User-agent: * Disallow: /*?utm_source* User-agent: * Disallow: /RestApi* User-agent: * Disallow: /rest-api*