> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usequotable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Site checks

> The daily check of each brand's site that finds what stops AI engines from reaching, reading or understanding it, and turns each failure into a Technical opportunity.

> [!NOTE]
>
> **Plans:** All plans · **In the app:** Sidebar → Actions → Opportunities

Quotable checks each monitored brand's site about once a day. Every check is a fixed rule over what your site sent back, with no AI model involved, so an unchanged site gives the same result every time and each finding points to the response, header or robots.txt line that caused it.

Each failing check becomes one item in the **Fix how AI reads your site** group on [Opportunities](https://docs.usequotable.com/actions/opportunities.md), filed under the **Technical** type. Passing checks are only counted.

## How the check runs

* **How often.** About once a day per brand. Brands that have never been checked go first.
* **Which brands.** Every monitored brand that has a domain. Paused brands and ended pitches are skipped, as they are for daily tracking.
* **What it requests.** About a dozen requests to the brand's own domain, and nowhere else:
  * robots.txt
  * up to three declared sitemaps, plus one sitemap inside a sitemap index
  * the homepage, fetched several ways: as a plain client, as an AI agent, a few times in a row for timing, and as Markdown
  * `/llms.txt`
  * one address that cannot exist, to see your "not found" response
* **From where.** From our network, under the name QuotableBot.

## Identify and allow QuotableBot

Most requests carry this user agent:

```text theme={"system"}
Mozilla/5.0 (compatible; QuotableBot/1.0; +https://usequotable.com)
```

One homepage request uses an AI-agent user agent to see how your firewall treats AI agents. It ends with `QuotableBot/1.0`, so your logs can tell it apart.

The site check reads your robots.txt and reports what it allows rather than following it, and it only requests your own domain. On other people's sites, QuotableBot follows robots.txt and answers to `User-agent: QuotableBot`. See [QuotableBot](https://docs.usequotable.com/trust/quotablebot.md) for how it behaves on any site.

> [!TIP]
>
> If your firewall challenges unknown bots, allow `QuotableBot` by user agent.
> Otherwise the checks report the challenge page rather than your site.

## What is checked

Impact is **High** when the problem stops engines from reaching or reading your site, or makes them read it badly. It is **Medium** when fixing it improves how they read you. Titles are shown as they appear on the board.

### Server and access

| Check | Passes when | Fails when | Impact |
| - | - | - | - |
| **Make *domain* answer with its homepage** | The homepage loads. | The homepage errors, times out or cannot be reached. | High |
| **Serve robots.txt as a plain text file** | robots.txt answers 2xx, or 404/410 (no rules). | It answers another error status or nothing. Crawlers read that as "stay out of the whole site." | High |
| | | It returns an HTML page instead of rules. | High |
| **Let AI search and answer bots read your site** | robots.txt lets these bots reach the site root: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, Amazonbot, DuckAssistBot and MistralAI-User. | Any of them is disallowed. The evidence names the rule and its line. | High |
| **Stop your firewall from challenging AI agents** | An AI agent gets the same homepage as a browser. | The homepage answers with a bot challenge, or answers an AI agent differently. | High |
| **Speed up the first response from your server** | The homepage's median response time is 1,000 ms or less. | It is slower than 1,000 ms. | High |
| **Say in robots.txt how AI may use your content** | robots.txt has a `Content-Signal` (or `Content-Usage`) line. | It has none, or there is no robots.txt. | Medium |

Blocking training crawlers such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent is your choice. The check leaves it alone.

### Indexing

| Check | Passes when | Fails when | Impact |
| - | - | - | - |
| **Answer 404 for pages that don't exist** | The made-up address answers any status outside 2xx. | It answers 200, or redirects to the homepage. | High |
| **Publish a sitemap and point robots.txt at it** | robots.txt declares sitemaps and each one can be read. | No sitemap is found anywhere. | High |
| **Fix the sitemaps robots.txt points to** | | A declared sitemap is missing, unreachable or not valid XML. | High |
| **Declare your sitemap in robots.txt** | | `/sitemap.xml` exists but robots.txt does not mention it. | Medium |
| **Give your sitemap real last-modified dates** | `lastmod` dates vary page by page. | There are none, some are more than a day in the future, 90% or more share one day, or most pages have none. | Medium |
| **Take the noindex tag off your homepage** | The homepage may be indexed. | It carries `noindex` in a robots meta tag or the `X-Robots-Tag` header. | High |
| **Put the homepage content in the HTML, not only in JavaScript** | The main text is in the HTML the server sends. | The HTML has little text before scripts run. | High |
| **Write a meta description for your homepage** | The homepage has one. | It has none. | Medium |
| **Give your homepage exactly one H1** | There is exactly one H1. | There is none, or more than one. | Medium |
| **Step your homepage headings down one level at a time** | No heading skips a level. | A heading jumps more than one level, such as H2 to H4. | Medium |

### Structured data

| Check | Passes when | Fails when | Impact |
| - | - | - | - |
| **Fix the structured data on your homepage that doesn't parse** | Every JSON-LD block parses, or there are none. | A JSON-LD block has a syntax error, so it is skipped whole. | High |
| **Declare who you are with Organization structured data** | The homepage declares an Organization (or a subtype such as Corporation, LocalBusiness or OnlineStore) with `name`, `url` and `logo`. | There is no Organization. | High |
| **Complete your Organization structured data** | | The Organization lacks `name`, `url` or `logo`. | High |
| **Link your Organization data to your official profiles** | The Organization has at least one `sameAs` link. | It has none. | Medium |
| **Add Open Graph tags to your homepage** | `og:title`, `og:type`, `og:image` and `og:url` are all present. | Any of them is missing. | Medium |

### Files for agents

| Check | Passes when | Fails when | Impact |
| - | - | - | - |
| **Publish an llms.txt file** | `/llms.txt` is published and has no errors. | It answers 404 or 410, or another error status. | Medium |
| **Serve llms.txt as Markdown, not as a web page** | | It returns an HTML page. | Medium |
| **Fix the errors in your llms.txt** | | It has format errors. The evidence lists up to five, by line. | Medium |
| **Serve Markdown to agents that ask for it** | Asking for the homepage with `Accept: text/markdown` returns `text/markdown`. | It still returns HTML. | Medium |

A check that lacks what it needs is skipped, not failed. For example, the heading checks are skipped when the homepage did not return HTML. Skipped checks do not count toward the total.

## Read a result

Open a Technical item to see:

* **What this is based on**: what the check found, such as the URL tested, the status it got or the rule that matched. The last line, **Checked**, is the day of the check.
* **The change**: for checks that can say exactly, what is there **Now** and what to **Change it to**. For example, your robots.txt group with a suggested `Content-Signal` line added, your heading outline corrected, or a starter llms.txt.
* **How to do it**: the steps to fix it.

With **Technical** as the only type filter and nothing failing, the list says "All *N* site checks passed." Before the first check it says "No site check has run yet."

## Crawl fixes from Agent Analytics

The site check shows what a visit *would* find. When the brand has an active [Agent Analytics](https://docs.usequotable.com/agent-analytics/overview.md) source, the same group also lists what AI systems *did* hit on real visits over the last 30 days:

| Item | What it means | Impact |
| - | - | - |
| **Unblock**, **Repair**, **Review crawler limits for** or **Stabilize** *path* | AI systems got a 401/403, a 404/410, a 429 or another error on this page. | High |
| **Agents fail across** *folder* | A whole section fails for AI systems more often than the rest of the site. | High |
| **Keep** *path* **reliable for agents** | A page already cited in AI answers is failing or slow for AI systems. | High |
| **Reduce latency on** *path* | The slowest 5% of AI visits to this page take 2 seconds or more. | Medium |
| **Review** *path*: **crawled, not cited** | AI systems fetch the page often and without errors, but no tracked answer cites it. | Medium |
| *path* **was not fetched by any agent** | The page is cited, but no AI system fetched it in the window. | Medium |

**Open the visits** takes you to the matching visits in Agent Analytics. These items leave the board on their own once later visits stop showing the problem.

## Fix and confirm

**1. Work on it with Quotable Agent**

Open the item and click **Work with agent**. [Quotable Agent](https://docs.usequotable.com/actions/quotable-agent.md) opens beside the page with the item loaded. It asks about your setup, such as your CDN or CMS, then writes the file, config or copy for your brand. You need edit access to the brand.

**2. Let the agent check the live page**

Quotable Agent reads pages on your brand's domain live: status, redirects, noindex, canonical, title, meta description, Open Graph tags, headings, files like robots.txt and llms.txt, and signs of a bot challenge or JavaScript-only content. It can also ask for a page as Markdown. It checks before it advises and again before it treats a fix as live. If it still sees the old version, it asks whether the change was published, since caches can lag.

**3. Mark it Done on the day it went live**

Set the status to **Done** and set **Done on** to the day the change went live. Quotable Agent can also propose the change, which you approve on a card. The [Impact](https://docs.usequotable.com/actions/impact.md) page puts its marker on that day.

**4. Wait for the next daily check**

The item leaves the board once a later check passes. There is no button to run the full site check on demand.

If the brand uses Agent Analytics, the **Crawlability** tab can also re-check robots.txt (**Check again**) and test one page (**Test page**) right away. See the [Agent Analytics dashboard](https://docs.usequotable.com/agent-analytics/dashboard.md).

## Limits

* The daily check looks at the site as a whole and at the homepage. It does not crawl every page.
* Quotable Agent checks up to six pages per reply, on the brand's domain and its subdomains only. Its checks share the brand's limit of 30 diagnostics per hour with the Agent Analytics tests.

## Next steps

> [card] [Opportunities](https://docs.usequotable.com/actions/opportunities.md)
>
> Work through the items with your team.

> [card] [Impact](https://docs.usequotable.com/actions/impact.md)
>
> See what moved after a fix went live.

> [card] [Agent Analytics](https://docs.usequotable.com/agent-analytics/overview.md)
>
> See what AI systems hit on real visits.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.