How to inventory bot visits and requested content

To identify which bots request your site and what they fetch, build a shared inventory from request logs, edge signals and bot classifications. Use it to distinguish observed requests from inferred identities, then prioritise pages where access problems could affect important content. Page analytics alone cannot answer the question reliably.

Which tools can measure which bots visit a site and what content they request?

Combine server access logs, edge or CDN and WAF records, bot-detection services, and crawler or indexing checkers to build an inventory of bot requests by URL and content type.

Each category answers a different part of the question:

Server access logs show requests that reach the origin server. They are useful for relating a requested path to a response status and content type.
Edge, CDN and WAF records show requests handled before they reach the origin. They can add context about filtering or challenges that an origin log will not contain.
Bot-detection services add classifications or behavioral signals. Treat a classification as evidence to inspect, particularly when an agent's identity is uncertain.
Crawler and indexing checkers help teams investigate discovery and indexing questions. They do not replace a record of requests to the site.

This combination is a practical way to compare what happened at the edge with what arrived at the origin. The approaches described by Olwen and Threat News illustrate why edge telemetry deserves a place alongside other request data.

Evaluate tools against the inventory you need to produce. Can your team export request-level records with a timestamp, path, user-agent string, IP address, response status and content type? Can it distinguish a request blocked at the edge from one served by the origin? Can reviewers inspect the evidence behind a bot label and correct false positives? A dashboard that reports a bot total but cannot connect it to requested URLs will not resolve which content those agents sought.

Googlebot, GPTBot, OAI-SearchBot and PerplexityBot are useful names to include when planning identification rules. Do not treat a matching user-agent string alone as proof of identity. Keep the observed string separate from the identity your team assigns.

Travatar is an AI-native analytics platform that monitors brand visibility in AI Search, classifies bot traffic and helps teams assess its impact on conversions. Its AI Visibility, AI Coworker and AI Traffic Analytics modules offer marketing, growth and product teams a way to connect visibility and traffic signals with work on identified gaps. An inventory still needs a defined request-data source and an agreed method for reviewing classifications.

How can a team build an inventory of bot visits and requested content?

Build the inventory by joining available request records to a consistent URL list, classifying agents with a stated confidence level, and grouping the resulting requests by content type.

Start with a bounded set of pages: for example, the pages your team uses to explain products, answer customer questions or support conversion. Include their canonical URLs and note whether important facts appear in rendered HTML. This gives marketing and engineering a common set of content to investigate rather than an undifferentiated traffic report.

Next, export request records for that scope from an origin or edge source. Preserve the raw fields before assigning bot labels. For each request, retain the time, requested path, user-agent string, IP address where available, response status and content type. If edge and origin records can be related, retain both observations so a blocked edge request is not mistaken for a page served successfully.

Then classify what was observed:

Separate clearly identified agents from ambiguous automated activity and ordinary traffic.
Record the evidence used for each classification, such as an operator-published identification method where one is available, or behavioral signals that warrant review.
Mark uncertain identities as uncertain. An unfamiliar user-agent string is not a named crawler merely because it requests many pages.
Review examples of each classification with the team responsible for traffic controls before using the labels to change access rules.

Finally, group requests by URL and resource type. Distinguish HTML pages from images, API responses and other assets. A request for an image associated with a page is different from a request for that page's HTML. Likewise, a request does not establish that an agent used the content in an AI answer or sent a visitor to the site.

Client-side JavaScript analytics should not be the sole input to this exercise. A request can appear in server or edge records without producing a client-side pageview. Compare the datasets to understand the difference in what they measure, rather than treating one total as a correction for the other. Where an agent's identity cannot be established from available identifiers, retain the request as observed activity with an unresolved classification.

What should a shared bot-by-content inventory contain?

A useful shared inventory links an observed agent or classification to a requested URL, a content type, a response outcome and the evidence behind the label.

Keep request observations and team decisions in the same working view, but in distinct fields. A compact row can contain:

Requested URL and canonical page, when the request concerns a page
Content type, such as HTML, image or API response
Observed user-agent string and assigned bot category
Classification confidence and supporting evidence
Request time or reporting period
Edge outcome, origin response status, or both when available
Responsible page owner and follow-up action

Example rows can describe the shape of the inventory without implying that a named bot visited your site:

Named crawler, HTML page: an observed request for a product explanation page, with its response status and the evidence used to identify the crawler. The page owner reviews whether the served HTML contains the intended facts.
Unresolved automated activity, image: an image request with an ambiguous user-agent string. The traffic owner reviews related requests before assigning an operator name.
Named crawler, blocked page: an edge record for a requested guide URL that did not reach the origin. The product and engineering teams review whether the access rule matches their intent.
Unresolved automated activity, API response: a request for a non-page resource. The team keeps it separate from HTML page visits when assessing content discovery.

The inventory should answer a precise question for every row: *What was requested, what happened to the request, and how certain are we about who made it?* That structure lets a marketing team identify affected content without asking engineering to accept unsupported bot labels. It also prevents a request count from being presented as a count of citations, recommendations or conversions.

How should teams prioritise bot-access findings and act on them?

Prioritise findings where a valuable page has a reproducible access or content-delivery problem, then assign a page owner and a measurable follow-up.

Begin with the observed outcome. A crawler requesting a page is not itself a problem. A relevant HTML page consistently returning an unintended error, being blocked contrary to policy, or omitting key facts from the served HTML gives the team a concrete issue to investigate. Keep uncertain classifications lower in the action queue until the traffic owner can verify them.

The next step depends on the symptom:

Blocked requests: review the applicable access rule and the intended policy for that agent. A declared permission is not proof that live requests can reach the page.
Unexpected response statuses or URLs: investigate redirects, canonicals and clean URL paths so the requested page resolves to the intended resource.
Key facts absent from rendered HTML: review whether client-only rendering limits what the relevant crawler receives. Test against that crawler's documented behavior rather than assuming every crawler works alike.
Important pages with weak discovery paths: add relevant internal links and use headings that state the questions the page answers. Use suitable Schema.org markup, such as entity, FAQ or Product markup, when the page content supports it; markup cannot substitute for missing content.
Slow or unstable responses: investigate performance when the request records show it is interfering with delivery.

For each change, record the affected URL, the before-and-after request outcome and the content or access issue resolved. Separately monitor AI visibility and conversion signals. A technical fix can make a page more accessible or its facts easier to extract; it does not guarantee a citation, position or conversion.

Marketing, growth and product teams can start by agreeing on a page scope and obtaining a request-level origin or edge export. Pair that export with bot-classification evidence, assign owners to the highest-impact findings, and use the next observation period to see whether the intended requests reach the intended content.

FAQ

Which log source should I start with when I only have permission for one?

Start with the request source that covers the URLs in scope and gives your team usable path, user-agent and response data. Server access logs show requests that reach the origin; CDN edge logs may also show requests stopped before the origin, so the better starting point depends on where access decisions occur.

How do I handle requests with ambiguous user-agent strings or missing IP ranges?

Keep the observed request and label the agent identity unresolved. Review related request behavior and any available operator documentation before assigning a named bot; an ambiguous string alone is insufficient evidence.

Can client-side analytics be relied on to count bot activity?

Client-side analytics should not be the only source for a bot inventory. Compare their pageview data with request-level server or edge records because the systems observe different events.

What minimal fields should a shared inventory row include?

Include the requested URL, content type, request time, observed user-agent, assigned category, classification evidence and response outcome. Add the responsible owner and follow-up action when a finding needs work.

How do I prioritise remediation when many bots fetch the same valuable pages?

Prioritise a reproducible problem on the page, such as an unintended block, failed response or missing facts in served HTML, rather than the number of bot labels alone. Assign the issue to the relevant page and traffic owners, then verify the request outcome after the change.

Which pages do AI crawlers request? Build a bot activity inventory