The 'Import Website' functionality allows you to schedule and perform recurring imports or crawls.
To enable it, go to Knowledge > New Folder > Import Website > Add the link to the webpage or sitemap > Select the Frequency.
To crawl the entire website on recurring basis, make sure to enable the 'Crawl Website' toggle.
Trigger Frequency:
Once: Import / crawl the website only once at the beginning.
Weekly: Import / crawl the website every 7 days. (Available on Advanced Plan)
Monthly: Import / crawl the website every 30 days. (Available on Premium and Advanced Plan)
Whenever recurring imports are enabled, the existing webpages in the folder will be updated with new content.
In case a particular webpage was deleted from the website, it will still remain in the Knowledge Base folder if it was part of a previous import or crawl.
Note: We recommend creating a new folder for recurring imports in case the existing folder contains duplicate webpages due to being crawled or imported twice before July 3rd 2024.