How to Scrape Website Data for Chatbot Training

• How to scrape website data using the Botvee Knowledge Base
• How to add a website URL for scraping
• Why having a sitemap improves the scraping process
• What happens if the website was already added before
• How to manage duplicate or replaced data

Log in to your Botvee dashboard and go to the Knowledge Base section. This is where you manage all chatbot training data including website pages, documents, and other knowledge sources.

Click the Add Website button and enter the full URL of the website you want to scrape (e.g., https://yourwebsite.com).

For the best results, your website should have a sitemap (e.g., https://yourwebsite.com/sitemap.xml). A sitemap helps Botvee find all important pages efficiently.

After adding the website URL, Botvee will automatically begin scraping — visiting website pages, extracting readable text content, removing unnecessary HTML or scripts, and converting the information into training data.

If the website has already been added before, you will see options:

• Replace — Replace the previously scraped content with new data
• Skip — Skip scraping if the website data already exists
• Duplicate — Add new scraped content while keeping the existing data

After scraping is complete, the extracted pages will appear in the Knowledge Base document list. Review to ensure important information was captured successfully.

Ask questions related to your website such as "What services do you offer?" or "How can I contact support?"

Website pages are not being scraped — Ensure the website URL is correct and publicly accessible. Some pages are missing — Check that the website sitemap includes those pages. Duplicate content appears in the knowledge base — Select Skip or Replace instead of Duplicate when re-scraping.