Find broken links and mixed content

The site crawl follows the links of a site once a day, checks every link it finds, and lists http:// resources on https pages (mixed content). You turn it on per uptime monitor; it is off by default.

Requirements

An uptime monitor that isn't paused.
At most 10 monitors with the site crawl per team, and 20 over all the teams an owner owns.
If a firewall, WAF or bot protection filters crawlers, allow the User-Agent lampwick-crawler.

Turn on the site crawl

Open the monitor's page from Uptime.
On the Site crawl: broken links & mixed content card, click Turn on site crawl. The first crawl starts within a minute or two.
If the monitor checks a health endpoint such as /up, open the results page (Crawl settings on the card, View results after the first crawl) and set Start crawl at to your homepage. Empty means the monitor URL.

Check that it works

After the crawl, the card shows Last crawl with the page count, the time it took and the counts of broken links and mixed content. View results opens the tabs Broken links and Mixed content, and Recent crawls (the latest 30).

What counts as broken

Result	Rule
Broken	Status 4xx or 5xx (except 429), a DNS, connection, TLS or timeout error, or more than 5 redirects.
Needs login	Status 401 or 403. Listed apart; not broken, no alert.
New	Broken or mixed since the previous successful crawl.

Broken links are grouped by 5xx, 404, other 4xx and connection errors, with a filter for Internal and External.

How the crawl works


Pages followed	`<a href>` links from the start URL to the same site: same scheme and port, `www.` and the bare domain as one host, judged from the start page's final URL after redirects.
Links checked	Every link found, external too. External links get `HEAD`, or a `GET` of at most 16 KB when `HEAD` fails, on ports 80 and 443 only.
Mixed content	Any `http://` subresource on an `https` page: `img` and `source` (with `srcset`), `script`, `link rel=stylesheet`, `iframe`, `video`, `audio`, `track`, `embed`, `object`, `url(...)` in an inline `style`. Listed once each, with up to 5 pages and the page count.
robots.txt	The group `lampwick`, otherwise `*`. Unreachable (5xx or timeout) means nothing is crawled. A page with `<meta name="robots" content="nofollow">` has its links skipped; `rel="nofollow"` links are checked, not followed.
Requests	User-Agent `lampwick-crawler/<version> (+https://lampwick.dev)`, one at a time per site, 250 ms apart.
Stored	URLs, status codes and Lampwick's own error text, never page content. Only the latest successful crawl's details are kept.

Limits per crawl

Limit	Value
Pages	250
Depth	10 clicks from the start page
Size	2 MB per page (only HTML is read), 50 MB in total
External links	500 links, 1,000 requests (every redirect counts), within 3 minutes
Time	10 seconds per request, 10 minutes in total
Slow site	Stops after 5 pages when they take more than 2 seconds each on average
Crawl now	Once per 10 minutes per monitor

A crawl that hits a limit says so, for example "Stopped early: the site has more than 250 pages". Turning the crawl on doesn't start the Crawl now wait.

Ignore a link

Ignore next to a link on the results page stops it counting and alerting; Unignore in the collapsed list brings it back. Up to 1,000 ignored links per monitor.

Alerts

After a crawl with new broken links or mixed content, the team gets an alert with counts only, no URLs:

Broken links: Shop The crawl of shop.example.com found 12 broken links (2 new) on 183 pages. Open the results to see where.

The first crawl reports everything it found. The same findings alert at most once per 30 days, and Mute alerts for everyone on the monitor mutes these alerts too.

API

getMonitorCrawl (settings, latest crawl, results), updateMonitorCrawl (enabled, start_url), startMonitorCrawl (crawl now; 429 within 10 minutes of the last one), ignoreCrawlLink and unignoreCrawlLink (body: url). Tokens and conventions: API.

Troubleshooting

What you see	Cause and fix
"Only the start page was crawled: it has no links to follow"	The monitor checks a health endpoint. Set Start crawl at to your homepage.
The crawl fails with a refused, reset or timed-out connection	The site blocks crawlers. Allow the User-Agent `lampwick-crawler` in your firewall or WAF.
The crawl fails on the certificate	Renew or fix the certificate, then use Crawl now.
robots.txt doesn't allow the start page	Add `User-agent: lampwick` and `Allow: /` to your robots.txt.
An external link is "not checked (non-standard port)"	Lampwick only checks external links on ports 80 and 443.
"Stopped early: the site responds slowly"	Pages took more than 2 seconds on average. Check your server's response time.