# Find broken links and mixed content

> Turn on the site crawl of an uptime monitor. Once a day Lampwick follows up to 250 pages, checks every link and reports http:// resources on https pages.

The site crawl follows the links of a site once a day, checks every link it finds, and lists `http://` resources on `https` pages (mixed content). You turn it on per [uptime monitor](/docs/uptime/); it is off by default.

## Requirements

- An uptime monitor that isn't paused.
- At most 10 monitors with the site crawl per team, and 20 over all the teams an owner owns.
- If a firewall, WAF or bot protection filters crawlers, allow the User-Agent `lampwick-crawler`.

## Turn on the site crawl

1. Open the monitor's page from **Uptime**.
2. On the **Site crawl: broken links & mixed content** card, click **Turn on site crawl**. The first crawl starts within a minute or two.
3. If the monitor checks a health endpoint such as `/up`, open the results page (**Crawl settings** on the card, **View results** after the first crawl) and set **Start crawl at** to your homepage. Empty means the monitor URL.

## Check that it works

After the crawl, the card shows **Last crawl** with the page count, the time it took and the counts of broken links and mixed content. **View results** opens the tabs **Broken links** and **Mixed content**, and **Recent crawls** (the latest 30).

## What counts as broken

| Result | Rule |
|---|---|
| Broken | Status 4xx or 5xx (except 429), a DNS, connection, TLS or timeout error, or more than 5 redirects. |
| **Needs login** | Status 401 or 403. Listed apart; not broken, no alert. |
| New | Broken or mixed since the previous successful crawl. |

Broken links are grouped by 5xx, 404, other 4xx and connection errors, with a filter for **Internal** and **External**.

## How the crawl works

| | |
|---|---|
| Pages followed | `<a href>` links from the start URL to the same site: same scheme and port, `www.` and the bare domain as one host, judged from the start page's final URL after redirects. |
| Links checked | Every link found, external too. External links get `HEAD`, or a `GET` of at most 16 KB when `HEAD` fails, on ports 80 and 443 only. |
| Mixed content | Any `http://` subresource on an `https` page: `img` and `source` (with `srcset`), `script`, `link rel=stylesheet`, `iframe`, `video`, `audio`, `track`, `embed`, `object`, `url(...)` in an inline `style`. Listed once each, with up to 5 pages and the page count. |
| robots.txt | The group `lampwick`, otherwise `*`. Unreachable (5xx or timeout) means nothing is crawled. A page with `<meta name="robots" content="nofollow">` has its links skipped; `rel="nofollow"` links are checked, not followed. |
| Requests | User-Agent `lampwick-crawler/<version> (+https://lampwick.dev)`, one at a time per site, 250 ms apart. |
| Stored | URLs, status codes and Lampwick's own error text, never page content. Only the latest successful crawl's details are kept. |

## Limits per crawl

| Limit | Value |
|---|---|
| Pages | 250 |
| Depth | 10 clicks from the start page |
| Size | 2 MB per page (only HTML is read), 50 MB in total |
| External links | 500 links, 1,000 requests (every redirect counts), within 3 minutes |
| Time | 10 seconds per request, 10 minutes in total |
| Slow site | Stops after 5 pages when they take more than 2 seconds each on average |
| **Crawl now** | Once per 10 minutes per monitor |

A crawl that hits a limit says so, for example "Stopped early: the site has more than 250 pages". Turning the crawl on doesn't start the **Crawl now** wait.

## Ignore a link

**Ignore** next to a link on the results page stops it counting and alerting; **Unignore** in the collapsed list brings it back. Up to 1,000 ignored links per monitor.

## Alerts

After a crawl with new broken links or mixed content, the team gets an alert with counts only, no URLs:

> **Broken links: Shop**
> The crawl of shop.example.com found 12 broken links (2 new) on 183 pages. Open the results to see where.

The first crawl reports everything it found. The same findings alert at most once per 30 days, and **Mute alerts for everyone** on the monitor mutes these [alerts](/docs/alerts/) too.

## API

`getMonitorCrawl` (settings, latest crawl, results), `updateMonitorCrawl` (`enabled`, `start_url`), `startMonitorCrawl` (crawl now; `429` within 10 minutes of the last one), `ignoreCrawlLink` and `unignoreCrawlLink` (body: `url`). Tokens and conventions: [API](/docs/api/).

## Troubleshooting

| What you see | Cause and fix |
|---|---|
| "Only the start page was crawled: it has no links to follow" | The monitor checks a health endpoint. Set **Start crawl at** to your homepage. |
| The crawl fails with a refused, reset or timed-out connection | The site blocks crawlers. Allow the User-Agent `lampwick-crawler` in your firewall or WAF. |
| The crawl fails on the certificate | Renew or fix the certificate, then use **Crawl now**. |
| robots.txt doesn't allow the start page | Add `User-agent: lampwick` and `Allow: /` to your robots.txt. |
| An external link is "not checked (non-standard port)" | Lampwick only checks external links on ports 80 and 443. |
| "Stopped early: the site responds slowly" | Pages took more than 2 seconds on average. Check your server's response time. |