CustomerChatBot
CustomerChatBot is the crawler behind CustomerChat.ai. It visits a website only when someone adds that website to their CustomerChat.ai chatbot, usually the business that owns it, so the chatbot can answer visitors' questions from the site's public pages.
What it does
- Identifies itself with the user agent
CustomerChatBot/1.0 (+https://customerchat.ai/bot). - Reads your sitemap and follows links within the same site, up to 150 pages at a time.
- Fetches a few pages at a time, follows your robots.txt crawl delay, and only reads public pages. It never logs in or submits forms.
- Also reads up to 20 PDF and Word documents your pages link to, the most linked-to first.
- Runs when the chatbot's owner adds the site or clicks Re-sync, and to keep the chatbot up to date: weekly, daily or hourly depending on their plan. Scheduled visits mostly happen overnight (UK time), follow your sitemap's last-updated dates, and only re-read pages that may have changed.
- If a page is blocked or only shows its content once JavaScript runs, the page may be fetched through Jina Reader, which loads it in a browser on CustomerChat.ai's behalf.
How to control it
CustomerChatBot follows the rules in your robots.txt. To keep it out of part of your site:
User-agent: CustomerChatBot Disallow: /private/
To block it from your whole site:
User-agent: CustomerChatBot Disallow: /
Rules under User-agent: * are followed too, except a site-wide Disallow: /. That's often left on by accident, and the person adding the site is usually its owner, so only a block that names CustomerChatBot stops it completely.
Questions
If CustomerChatBot is causing problems on your site, or you'd like it to stop, please get in touch.