In this help article, we will guide you through the process of using the website crawler feature in Cody to import content from your website.
Step 1: Navigate to the Content Tab
Log in to your Cody account and go to the Content Tab.
Step 2: Create a Folder
Click on the "Create Folder" button to create a new folder for storing the imported content from your website.
Step 3: Select Import from Website
Select the newly created folder and click on the "Import from Website" button.
Step 4: Begin Importing
As a Free User, you can import a single URL one at a time. As a Premium User, you can crawl an entire website for content.
The website crawler will crawl your website for all textual content that Cody can learn from. It works even better if you import with a sitemap link. The crawler can crawl up to 5,000 pages.
If you want to import only a certain category of webpages, insert the URL for that category, and it will be considered as the seed URL.
For Example:
https://dundermifflin.com/employees has multiple pages such as:
https://dundermifflin.com/employees/Michael
https://dundermifflin.com/employees/Jim
https://dundermifflin.com/employees/Dwight
Cody will import all webpages under the "employees" pathname viz. Michael, Jim, and Dwight.
How do I stop an import?
You can stop a website from being crawled by clicking on the 'stop' icon on the right side of the import progress-bar.
If you stop the crawler mid-way, you will be provided with two options:
"Stop but store indexed pages" allows you to save the pages that have already been crawled and indexed.
"Stop and don't store any pages" will discard any page that has been indexed.
Troubleshooting
If you're having issues with the website crawler, follow these recommendations:
Make sure your URL includes "https://" or "http://" at the beginning.
Create a new folder specifically for the website crawl.
If the crawl doesn't work initially, wait for 5 minutes and try again to see if it goes through.
By following these steps, you can successfully import content from your website into Cody's knowledge base, allowing it to provide more accurate and relevant information based on your website's content.