Agents

Add a web page or crawl a site

Point the library at a public HTTPS URL, choose a single page or a crawl, and re-crawl it when the content changes.

A web source points the library at a URL instead of a file. You can index a single page, or crawl a site and index everything the crawl reaches. Either way you get a snapshot, which you refresh when the site changes.

What you need

  • A workspace you are a member of.
  • A public HTTPS URL. Plain HTTP, loopback addresses, private ranges and link-local addresses are rejected.
  • A page that serves its content as HTML. Content that only appears after JavaScript runs may not be reachable.

Steps

Step 1: Open the add flow

Click Add in Knowledge base, or Add New on an agent's Knowledge tab, then choose Add Link.

Step 2: Choose single page or crawl

The dialog opens on Single Page, which indexes exactly the URL you give it. Switch to Crawl site to follow links from that URL as well.

Step 3: Enter the URL

Fill URL with the full address, starting with https://.

Step 4: Set the crawl bounds

These three fields only appear under Crawl site.

Field Portal starting value Accepted range
Depth 2 1 to 5
Max pages 100 1 to 500
URL pattern empty Up to 500 characters

Depth is how many link hops from the seed URL the crawl may travel. Max pages is the hard ceiling on how many pages it fetches. URL pattern restricts which links are followed; leave it empty to follow everything the crawl finds.

Step 5: Choose where it lands

Under Location, leave Library root to keep the source loose, or pick a folder.

Step 6: Add it

Click Add.

Get it right

Bound the crawl at the section you want. A crawl of a whole site fills the library with navigation pages and blog posts that compete with the page the caller was actually asking about.

Recommended

URL:        https://acme.example/help/returns
Depth:      1
Max pages:  25

Less effective

URL:        https://acme.example
Depth:      5
Max pages:  500

Start with Depth and Max pages. Add a URL pattern only when you know the pattern the section's links follow, and check the page count on the finished source to see what it caught.

Worked example

You add https://acme.example/help/returns as a crawl with Depth 1 and Max pages 25, into the Returns folder. The row goes Pending, then Crawling, then Indexing, and reaches Ready with 9 pages and 61 chunks, along with a Web source indexed notification.

Two months later the returns window changes. You open the row menu in Knowledge base, choose Refresh, and the same URL is crawled again with the same settings. The new result replaces the old one, and every agent that selected the Returns folder uses it on the next conversation.

What happens next

The source is created at Pending, then moves through the crawl and the indexing:

Status Portal label What it means
pending Pending Created and waiting to start.
crawling Crawling Fetching pages.
processing Indexing Fetching is done and the text is being indexed.
ready Ready Finished. The page count and chunk count are filled in.
failed Failed The crawl or the indexing did not finish. The row carries the reason.

When it finishes you get a notification: Web source indexed, naming the URL with its page and chunk counts, or Web source failed to index with the reason.

What is kept is the extracted text of the pages, split into chunks, plus the page count. The original HTML is not kept, so a web source has no Download action.

A source that stays at Crawling or Indexing for more than 60 minutes fails with Indexing timed out. Try again.

Re-crawling

Websites change, and a web source is a snapshot of the day it was crawled. To bring it up to date, open Knowledge base, open the row menu on the web source and choose Refresh. It crawls the same URL again with the same settings: the source goes back to Crawling, the refresh time is recorded, and the result replaces what was indexed before. Refresh is also the recovery path for a failed web source.

Refresh it by hand whenever the page matters and the site has changed.

The row menu on an agent's Knowledge tab covers that agent's selection only, so Refresh lives in the library.

Limits

Limit Value Applies to
Scheme HTTPS only Every web source
Host No loopback, private or link-local addresses Every web source
URL length 2000 characters The create request
Display name 200 characters The create request and renaming
Depth 1 to 5 Crawls
Max pages 1 to 500 Crawls
URL pattern 500 characters Crawls
Indexing timeout 60 minutes Crawling and indexing

Web sources do not count against your plan's storage allowance.

Troubleshooting

Use a public URL that starts with https://. Private or localhost addresses are rejected.

The portal's own check. The address looks like localhost, a loopback address, or a private range such as 192.168.x.x or 10.x.x.x.

The address must start with https://. The dialog accepts an http:// address and the request is then refused. Change the scheme to https:// and try again.

Refresh is refused on a source that never crawled

Refresh re-runs a crawl that started at least once. A source still at Pending, or one that failed before it started, has nothing to re-run: delete it and add it again.

Indexing timed out. Try again.

The crawl ran for over an hour without a result. Try a smaller crawl: lower Depth, lower Max pages, or narrow it with URL pattern.

Ready, but the page count is 1 on a site crawl

The crawl found no links it was allowed to follow. Check the seed URL really links out, and that URL pattern is not excluding everything.

Ready, but the chunk count is very low

Little text was extracted. The most common cause is a page that renders its content with JavaScript, leaving almost nothing in the HTML that is fetched. Point at a server-rendered page, or paste the text in as a text source instead.

FAQ

Does a crawl stay in sync with the site?

No. A crawl is a snapshot. Use Refresh when the site changes.

Can I crawl a page behind a login?

No. The URL is fetched with no credentials of yours, so only publicly reachable pages work.

What happens to the old pages when I refresh?

The refresh re-crawls the same URL with the same settings, and the new result replaces the old one for that source.

Can I add many URLs at once?

Not in one action. Add each URL as its own source, or point a crawl at a section of the site and bound it with Depth and Max pages.

Does a web source use storage?

No. Only files and pasted text count against your plan's storage allowance.