Your website serves more than the people who visit it. Search engines and AI services may also retrieve its pages. The useful question is what you want to make available, to whom, and for what purpose.
For a business, that means separating three decisions: helping customers find public information, setting preferences for how AI providers use it, and protecting anything that should stay private. Each needs a different approach.
Be found
Clear public pages, useful links and search settings.
Visibility is an opportunity, not a guarantee.Set preferences
Provider-specific rules for search and model training.
Rules depend on the provider following them.Protect access
Authentication and permissions for private information.
Crawler instructions cannot replace security.Start with what customers should find
Decide which information helps someone understand your business: services, expertise, contact details and useful explanations. Keep those pages accurate and easy to navigate. A clear answer on your website is valuable whether someone reaches it through a search engine, an AI assistant or a direct link.
Being available to a crawler does not guarantee that an AI service will mention your business, represent it accurately or send visitors to your site. Check how your organisation appears in relevant searches, and correct unclear or outdated information at the source.
Search and training are different uses
Some providers offer separate controls. OpenAI, for example, documents OAI-SearchBot for search and GPTBot for content that may be used in model training. Its settings allow a website owner to permit the former while disallowing the latter. User-triggered visits are a separate case and may not follow the same crawler rules. Read OpenAI’s crawler guidance.
That is a provider-specific arrangement, not a universal switch for every AI tool. Before changing your settings, ask your website provider which services the rules cover and how they will check that the configuration works. Do not assume a new preference removes material already collected or used.
Keep private information behind access controls
A robots.txt file tells cooperating crawlers where they may go. It does not prevent someone from opening a page, and a blocked URL can still appear in Google’s results. Private documents and internal systems need authentication and appropriate permissions. Google explains these limitations.
Removing a public page from search is a separate task. Google must be able to crawl a page to see a noindex instruction. Blocking that crawl can stop it reading the instruction. If confidential information has become public, secure or remove it immediately and then address search results; do not leave it exposed while waiting for a crawler. See Google’s indexing guidance.
A practical review for your business
You do not need to start with a large programme. Bring the person responsible for your website together with your IT lead or provider and agree four things:
- Map what is public.
Include documents, old pages, subdomains and test environments. Identify anything that should require a login.
- Agree the intended use.
Record what should be discoverable and which provider-specific training preferences you want to apply.
- Check the implementation.
Ask for evidence that private areas require authentication and that crawler and indexing settings match the agreed decisions.
- Assign an owner.
Review the settings when the website changes and as providers update their guidance. Use available logs and search tools to check the results.
The outcome should be a short record of decisions, named responsibilities and any changes needed. That gives management something concrete to review without having to manage every technical setting themselves.