Skip to content

Odduel

We put the odds in your favour

Insights

How to prepare a website for AI search: a practical checklist

A practical checklist to prepare a website for AI search: crawler policy, llms.txt, structured data, answer-first content, technical basics and how to measure visibility in AI answers.

A web developer seen from behind at a standing desk by a window over European rooftops, preparing a website's structure for AI search

To prepare a website for AI search, make sure AI crawlers are allowed to read it, that every important page answers a clear question in plain, quotable text, and that structured data, clean headings and consistent company facts tell machines exactly who you are and what you offer. The same foundations that help classic search, such as fast pages, indexable content and trustworthy sources, are what AI Overviews, ChatGPT and other assistants draw on when they cite a site. This checklist covers crawler policy, llms.txt, structured data, content, technical basics and measurement, in the order we work through them.

Last updated: 14 November 2026.

What does AI search change for a website?

AI search is any search experience where an AI model writes an answer and cites sources, rather than only listing links. Examples include Google’s AI Overviews and AI Mode, answer engines and chat assistants that browse the web. For a website, the change is simple to state: being one of the sources an answer quotes now matters as much as ranking for the blue links.

The good news is that there is no separate rulebook. Google states in its guidance on AI features and your website that the usual best practices for Search apply, and that no special markup is required to appear in AI Overviews. What changes is the emphasis: clear answers, credible facts and pages that machines can read without effort.

At Odduel, a creative and technology agency in Barcelona and São Paulo, we build these foundations into every site in our website development service, and we apply them to our own site.

In short: allow the crawlers you want, publish a sensible llms.txt, add accurate structured data, write each key page around a question with a direct answer at the top, keep company facts identical everywhere, make pages fast and indexable, then track citations and referrals over time.

How do you appear in AI Overviews?

To appear in AI Overviews, a page must first be indexed and eligible to show a snippet in Google Search. Beyond that, the pages that tend to be cited share a few traits:

  • They answer the question directly, in the first sentences, without a long introduction.
  • They cover the topic in depth, including the follow-up questions people ask next.
  • They are clearly structured, with descriptive headings, short paragraphs, lists and tables.
  • They are trustworthy: a known author or company, sources for facts, a recent update date.
  • They are technically sound: fast, mobile-friendly, without blocking scripts or hidden text.

There is no trick that guarantees a citation. Pages that are the best answer for a question, and easy to read and verify, are the ones most likely to be quoted.

Here is the checklist, in the order we apply it.

  1. Decide your crawler policy. List the AI crawlers you want to allow for search and answers, and those you may want to block from training. Write the rules in `robots.txt`.
  2. Check indexing. Make sure important pages are not blocked by `robots.txt`, `noindex` tags or login walls, and that they appear in your XML sitemap.
  3. Publish llms.txt. Add a short Markdown file at `/llms.txt` that describes the site and links to its most useful pages.
  4. Add structured data. Organization, WebSite, BreadcrumbList, Article, Product, Service, FAQPage and others where they truly match the content.
  5. Rewrite key pages around questions. One main question per page, a direct answer first, then detail.
  6. Make facts consistent. Company name, description, address policy, services and profiles should match on the website, in structured data and on external profiles.
  7. Strengthen technical basics. Fast loading, server-rendered content, clean HTML, descriptive alternative text and working internal links.
  8. Earn references. Mentions in reputable publications, directories and partner sites help AI systems trust what you say about yourself.
  9. Measure and iterate. Track AI referrals in analytics, check how assistants describe you, and update pages that are outdated.

What should your crawler policy say?

Your `robots.txt` file tells crawlers what they may read. Many AI companies run more than one crawler: one used to fetch pages for answers and search, and another used to collect training data. They can often be allowed or blocked separately. For example, OpenAI documents GPTBot for training and OAI-SearchBot for search, and Google offers the Google-Extended token to control use of content for its AI models without affecting Google Search.

A typical policy for a business that wants visibility is:

  • Allow search engines and the crawlers that power AI answers and citations.
  • Decide deliberately whether to allow training crawlers; blocking them does not remove you from search.
  • Keep private areas, admin pages and duplicate parameters blocked for everyone.
  • Review the list of crawlers once or twice a year, because names and purposes change.

Remember that `robots.txt` is a request that reputable crawlers follow, not a security barrier. Private content must be protected by sign-in.

What is llms.txt?

llms.txt is a proposed standard, described at the llms.txt proposal, for a Markdown file placed at the root of a website that gives AI models a short, structured summary of the site: what it is, and links to the most useful pages with a line about each. The idea is to help models find clean, relevant content without crawling every page.

Support for llms.txt varies between platforms and is not confirmed by all of them, so treat it as a low-cost addition rather than a ranking factor. A good file is short, accurate and generated from your published pages so it never goes out of date. We publish one on our own site.

SignalWhat it isWhat it helps withEffort
robots.txtRules for which crawlers may read which pathsAccess for search and AI crawlersLow
XML sitemapList of indexable addressesDiscovery of pagesLow
llms.txtMarkdown summary of the site and key pagesOrientation for AI models where supportedLow
Structured dataMachine-readable facts in JSON-LDUnderstanding of entities, products, articles, FAQsMedium
Answer-first contentPages written around clear questionsBeing quoted in answersMedium to high
The main signals that prepare a website for AI search, and what each one does.

Which structured data matters most?

Structured data in JSON-LD tells machines exactly what a page contains. For most business sites, start with:

  • Organisation details: name, logo, description, contact points and official profiles through `sameAs`.
  • WebSite and BreadcrumbList: the site’s structure and each page’s place in it.
  • Article for blog posts and guides, with author, dates and headline.
  • Service or Product for what you sell, with accurate descriptions.
  • FAQPage where a page genuinely contains questions and answers.

Structured data must match what is visible on the page. Marking up content that is not there, or inventing reviews and ratings, can lead to manual actions and damages trust. Our guide to breadcrumb navigation shows one small, useful piece of this in practice.

How should content be written for AI answers?

AI systems quote passages that stand on their own. Write so that any paragraph could be lifted into an answer and still make sense:

  • Lead with the answer. The first two sentences of a page should answer its main question completely.
  • One idea per paragraph, with the key sentence first.
  • Use questions as headings where natural, phrased the way buyers ask them.
  • Define terms clearly: “X is…” sentences are easy to quote.
  • Cite sources for figures and dates, and keep them current.
  • Show who is behind it: company details, authors and an update date.
  • Avoid vague marketing language; concrete statements are more likely to be used.

If you are planning a redesign at the same time, our guide on SEO during a website redesign explains how to protect what already ranks.

How long does it usually take?

The technical part, covering crawler policy, sitemap, llms.txt and core structured data, can be done in days on a well-built site. Rewriting key pages around questions takes longer and depends on how many pages matter: start with the home page, the service or product pages and the most visited articles. Results then build over weeks and months as crawlers revisit pages and as other sites refer to you. Treat it as continuous work, reviewed alongside your normal SEO and content calendar, and run a website QA checklist after each round of changes.

How do you measure AI search visibility?

Measurement is still maturing, but several signals are useful:

  • Referrals from AI assistants and answer engines in your analytics.
  • Search Console data for queries and pages, since AI Overviews traffic is included in Search reporting.
  • Manual checks: ask the main assistants the questions your buyers ask and note whether and how you are cited.
  • Brand descriptions: check that assistants describe your company accurately, and fix inconsistent facts at the source.

Record a baseline before changes, then review monthly.

Questions people ask

What is llms.txt?

llms.txt is a proposed standard for a Markdown file at the root of a website that summarises the site for AI models and links to its most useful pages with short descriptions. Support varies between platforms, so treat it as a low-cost addition that helps where it is used, not as a ranking factor.

How do you appear in AI Overviews?

Your page must be indexed and eligible for a snippet in Google Search. Beyond that, pages that answer a question directly, cover the topic in depth, use clear structure, show who is behind them, cite sources and load quickly are the ones most likely to be cited. No special markup is required.

Should I block AI crawlers in robots.txt?

It depends on your goals. Blocking crawlers that power AI answers can remove you from citations, while blocking training crawlers usually does not affect search. Many businesses allow search and answer crawlers and decide separately about training. Review the crawler list regularly, as names and purposes change.

Does structured data help with AI search?

Yes, when it is accurate. JSON-LD for your organisation, articles, products, services, breadcrumbs and genuine FAQs helps machines understand who you are and what each page contains. It must match the visible content; invented reviews or markup for content that is not on the page can harm trust and visibility.

How long does it take to prepare a website for AI search?

The technical basics, such as crawler rules, sitemap, llms.txt and structured data, can take days on a well-built site. Rewriting key pages around clear questions takes longer. Visibility then builds over weeks and months as crawlers revisit pages and other sites reference yours, so treat it as continuous work.

Open your `robots.txt` and your home page today: check who you allow and whether the first two sentences answer what you do. If you want a website built to be read and cited by search engines and AI assistants, see our website development service or tell us about your site.

More articles

More from our Insights.

Explore all insights

Get in touch

Let’s change your odds.

Odds do not change on their own. If you have something to build, launch or fix, tell us about it and we will show you how we would approach it.

    Need an NDA before you share details? Start a project

    Chat with us