Sometimes, when you try to add a webpage to your Agent’s Knowledge Base using a URL, the content may not load or scrape correctly.
This can happen for a few reasons related to how certain websites are built or protected.
Why It Happens
Some webpages include security measures or configurations that block automated scraping tools. This is often done to protect content or control how it’s accessed. Common blockers include:
CORS (Cross-Origin Resource Sharing): Prevents content from being loaded by outside systems.
CAPTCHAs or Login Walls: Require human interaction before showing content.
JavaScript-Heavy Pages: Content may load dynamically after the initial page request, making it difficult for scrapers to access.
Content Protection Plugins: Some sites intentionally block scraping bots.
Recommended Workaround
If you run into issues with scraping a URL, don’t worry, you can still add that content to your agent.
Here's what to do instead:
Manually copy the content from the webpage (e.g., FAQs, service descriptions, product info).
Paste the content into ChatGPT or another AI system with a prompt like:
“Convert this into a structured knowledge base format for an AI agent. Use an FAQ-style format with clear questions and answers.”Review and refine the output, then copy it into your agent’s Knowledge Base manually.
📝 Best Practices for Formatting
To get the most useful responses from your AI agent
Use FAQ-style formatting:
Q: What are your business hours?
A: We are open from 9 am to 6 pm Monday to Friday.
Q: How can I book an appointment?
A: You can book online through our website or call us directly.
Break down long paragraphs into smaller Q&A pairs.
Include common questions you expect users to ask.
💬 Need Help?
If you're still having trouble or unsure how to format your content, please contact us at hello@salescloser.ai or visit our website.
Our support team will be happy to help guide you!
