> For the full page index, fetch https://echonova.ai/llms.txt # Add a web page or crawl a site Point the library at a public HTTPS URL, choose a single page or a crawl, and re-crawl it when the content changes. A web source points the library at a URL instead of a file. You can index a single page, or crawl a site and index everything the crawl reaches. Either way you get a snapshot, which you refresh when the site changes. ## What you need - A workspace you are a member of. - A public HTTPS URL. Plain HTTP, loopback addresses, private ranges and link-local addresses are rejected. - A page that serves its content as HTML. Content that only appears after JavaScript runs may not be reachable. ## Steps ### Step 1: Open the add flow Click **Add** in **Knowledge base**, or **Add New** on an agent's **Knowledge** tab, then choose **Add Link**. ### Step 2: Choose single page or crawl The dialog opens on **Single Page**, which indexes exactly the URL you give it. Switch to **Crawl site** to follow links from that URL as well. ### Step 3: Enter the URL Fill **URL** with the full address, starting with `https://`. ### Step 4: Set the crawl bounds These three fields only appear under **Crawl site**. | Field | Portal starting value | Accepted range | | --------------- | --------------------- | -------------------- | | **Depth** | 2 | 1 to 5 | | **Max pages** | 100 | 1 to 500 | | **URL pattern** | empty | Up to 500 characters | **Depth** is how many link hops from the seed URL the crawl may travel. **Max pages** is the hard ceiling on how many pages it fetches. **URL pattern** restricts which links are followed; leave it empty to follow everything the crawl finds. ### Step 5: Choose where it lands Under **Location**, leave **Library root** to keep the source loose, or pick a folder. ### Step 6: Add it Click **Add**. ## Get it right Bound the crawl at the section you want. A crawl of a whole site fills the library with navigation pages and blog posts that compete with the page the caller was actually asking about. **Recommended** ```text URL: https://acme.example/help/returns Depth: 1 Max pages: 25 ``` **Less effective** ```text URL: https://acme.example Depth: 5 Max pages: 500 ``` Start with **Depth** and **Max pages**. Add a **URL pattern** only when you know the pattern the section's links follow, and check the page count on the finished source to see what it caught. ## Worked example You add `https://acme.example/help/returns` as a crawl with **Depth** 1 and **Max pages** 25, into the **Returns** folder. The row goes **Pending**, then **Crawling**, then **Indexing**, and reaches **Ready** with 9 pages and 61 chunks, along with a **Web source indexed** notification. Two months later the returns window changes. You open the row menu in **Knowledge base**, choose **Refresh**, and the same URL is crawled again with the same settings. The new result replaces the old one, and every agent that selected the **Returns** folder uses it on the next conversation. ## What happens next The source is created at **Pending**, then moves through the crawl and the indexing: | Status | Portal label | What it means | | ------------ | ------------ | --------------------------------------------------------------------- | | `pending` | **Pending** | Created and waiting to start. | | `crawling` | **Crawling** | Fetching pages. | | `processing` | **Indexing** | Fetching is done and the text is being indexed. | | `ready` | **Ready** | Finished. The page count and chunk count are filled in. | | `failed` | **Failed** | The crawl or the indexing did not finish. The row carries the reason. | When it finishes you get a notification: **Web source indexed**, naming the URL with its page and chunk counts, or **Web source failed to index** with the reason. What is kept is the extracted text of the pages, split into chunks, plus the page count. The original HTML is not kept, so a web source has no **Download** action. A source that stays at **Crawling** or **Indexing** for more than 60 minutes fails with `Indexing timed out. Try again.` ## Re-crawling Websites change, and a web source is a snapshot of the day it was crawled. To bring it up to date, open **Knowledge base**, open the row menu on the web source and choose **Refresh**. It crawls the same URL again with the same settings: the source goes back to **Crawling**, the refresh time is recorded, and the result replaces what was indexed before. **Refresh** is also the recovery path for a failed web source. Refresh it by hand whenever the page matters and the site has changed. The row menu on an agent's **Knowledge** tab covers that agent's selection only, so **Refresh** lives in the library. ## Limits | Limit | Value | Applies to | | ---------------- | -------------------------------------------- | ------------------------------- | | Scheme | HTTPS only | Every web source | | Host | No loopback, private or link-local addresses | Every web source | | URL length | 2000 characters | The create request | | Display name | 200 characters | The create request and renaming | | **Depth** | 1 to 5 | Crawls | | **Max pages** | 1 to 500 | Crawls | | **URL pattern** | 500 characters | Crawls | | Indexing timeout | 60 minutes | Crawling and indexing | Web sources do not count against your plan's storage allowance. ## Troubleshooting ### Use a public URL that starts with https://. Private or localhost addresses are rejected. The portal's own check. The address looks like `localhost`, a loopback address, or a private range such as `192.168.x.x` or `10.x.x.x`. ### url must be a public HTTPS URL (no loopback, private, or link-local addresses) The address must start with `https://`. The dialog accepts an `http://` address and the request is then refused. Change the scheme to `https://` and try again. ### Refresh is refused on a source that never crawled **Refresh** re-runs a crawl that started at least once. A source still at **Pending**, or one that failed before it started, has nothing to re-run: delete it and add it again. ### Indexing timed out. Try again. The crawl ran for over an hour without a result. Try a smaller crawl: lower **Depth**, lower **Max pages**, or narrow it with **URL pattern**. ### Ready, but the page count is 1 on a site crawl The crawl found no links it was allowed to follow. Check the seed URL really links out, and that **URL pattern** is not excluding everything. ### Ready, but the chunk count is very low Little text was extracted. The most common cause is a page that renders its content with JavaScript, leaving almost nothing in the HTML that is fetched. Point at a server-rendered page, or paste the text in as a text source instead. ## FAQ ### Does a crawl stay in sync with the site? No. A crawl is a snapshot. Use **Refresh** when the site changes. ### Can I crawl a page behind a login? No. The URL is fetched with no credentials of yours, so only publicly reachable pages work. ### What happens to the old pages when I refresh? The refresh re-crawls the same URL with the same settings, and the new result replaces the old one for that source. ### Can I add many URLs at once? Not in one action. Add each URL as its own source, or point a crawl at a section of the site and bound it with **Depth** and **Max pages**. ### Does a web source use storage? No. Only files and pasted text count against your plan's storage allowance. ## Related - [How knowledge works](/docs/agents/knowledge-overview): what reaches a turn, and when - [Add documents to the library](/docs/agents/knowledge-documents): upload the page as a file instead - [Add text directly](/docs/agents/knowledge-text): paste the text when the page renders with JavaScript - [Organize knowledge in folders](/docs/agents/knowledge-folders): keep a site's pages together - [The agent did not use my document](/docs/agents/knowledge-troubleshooting): when a ready source goes unused