AI training and grounding preferences are splitting into separate web publisher controls on Microsoft Bing. Site owners no longer have to treat search visibility and generative model ingestion as an all-or-nothing trade-off. As detailed in an update announced on the Bing blog by Krishna Madhavan, Principal Product Manager at Microsoft Bing, publishers can configure distinct settings for generative model training and live response grounding while remaining eligible for traditional organic search results.
These changes build on work with the Internet Engineering Task Force (IETF) AI Preferences working group. Microsoft is introducing standardized syntax for robots.txt files and page-level directives that replace older legacy tags like NoArchive and NoCache. Website administrators and digital businesses need to prepare their technical rules ahead of the enforcement deadline.
The Core Changes in Microsoft Publisher Controls
In 2023, Microsoft became the first search engine to provide webmasters with publisher controls for artificial intelligence training and grounding by using NoArchive and NoCache robots tags. While those tags offered an initial layer of restriction, they tied multiple search engine features together and lacked flexibility for emerging generative interfaces.
Under the announced system, Microsoft isolates AI training from real-time grounding. Both choices default to Allow if no explicit preference is published, but each preference operates independently. Publishers evaluating their AI training and grounding preferences need to review both site-level files and page templates to establish clear boundaries for their assets.
- AI Training controls whether your content is used to modify the learned parameters of generative models. This preference is expressed globally or by path in robots.txt, with optional page-level overrides.
- AI Grounding controls whether web content is referenced during live answer synthesis in Copilot, Bing AI Mode, AI Overview, and Microsoft 365. This preference operates exclusively at the page level.
- DisallowExtended is an added grounding setting that allows Microsoft-operated products to reference pages while excluding third-party grounding syndication via Foundry APIs and Web IQ.
- Eligibility for traditional search ranking remains unaffected regardless of whether training or grounding is blocked.
How Independent AI Training and Grounding Preferences Work
Understanding the distinction between these two functions prevents accidental loss of audience exposure. Model training involves ingesting text corpora to update neural weights. In contrast, grounding acts as a retrieval mechanism: a generative interface pulls real-time information to formulate an answer, citing the original publisher with links.
Because the AI training and grounding preferences operate on distinct layers, a publisher can prevent future model training while keeping content available for retrieval citations. For instance, setting training to Disallow while setting grounding to Allow keeps brand URLs eligible to appear inside Copilot summaries. Publishers choosing Disallow or DisallowExtended for grounding will experience reduced brand visibility within AI summary interfaces, as those engines are barred from using the page content to formulate direct answers.
Under the IETF working definition adopted by Microsoft, AI training specifically covers using an asset to modify the learned parameters of a generative model. Training models that perform exclusively non-generative tasks is not included in this category, even if the underlying model architecture possesses generative capabilities. Furthermore, traditional search indexing, page ranking, and standard web snippets do not constitute grounding.
Robots.txt Syntax and Page Directives
Microsoft adopted the IETF AI Preferences working draft for machine training preferences. To manage whole-site training restrictions, site owners add a designated user-agent block within their robots.txt file. A full-site exclusion uses standard disallow lines targeting the training identifier:
User-Agent: __usage_ai-training
Disallow: /
Path-specific rules function similarly, allowing publishers to exclude specific folders, such as private documentation or paid resource directories, by replacing the root slash with a targeted path like /example-path/. If a site administrator wants a specific page to opt back into model development despite a site-wide block, a page-level AI training tag overrides the broader robots.txt instruction.
Grounding rules do not live inside robots.txt. Instead, they require page-level meta directives. Setting grounding to Disallow completely blocks live generative summaries from referencing that URL. Selecting DisallowExtended creates a selective boundary: Microsoft internal tools can cite the material, but external developers accessing Microsoft grounding endpoints cannot pull from it.
When configuring AI training and grounding preferences across complex domains, engineering teams must remember that traditional search APIs operate under separate channels. The current Bing search syndication API returns links for standard search scenarios and does not power generative summaries, meaning it remains unaffected by page-level grounding rules.
Key Implementation Dates for AI Training and Grounding Preferences
Microsoft outlined a defined timeline for rolling out these AI training and grounding preferences. Starting February 1, 2027, Microsoft crawler systems will begin honoring the new preferences. If crawlers detect the new syntax on a domain, older NoArchive and NoCache flags are disregarded immediately.
Site owners relying on legacy protections must update their infrastructure by March 1, 2027. After March 1, 2027, Microsoft will no longer read NoArchive or NoCache as an instruction for model training or grounding. For grounding rules during the transition, Microsoft requires webmasters to publish page tags and record a temporary transition preference inside Bing Webmaster Tools. This temporary bridge safeguards pages until they are recrawled, or until September 30, 2027, whichever comes first.
Publishing these choices does not create a formal licensing contract or guarantee monetary compensation. Allowing training helps improve future models, and allowing grounding keeps URLs eligible for source citations, but neither setting guarantees inclusion or traffic.
Practical Impact on Enterprise Web Assets
For brands and digital agencies managing custom CMS deployments, these protocol changes require planned audits. Web development workflows need to incorporate metadata checks during page publishing cycles. Those seeking structured technical design can align updates through web designing standards that account for evolving search protocols.
Automated content operations must also take heed. If your organization distributes programmatic assets or manages knowledge hubs via AI automation, failing to update outdated caching tags could accidentally open closed content libraries to training datasets after March 2027. Reviewing robots.txt delivery rules across staging and production environments ensures configurations remain compliant.
Large digital properties often serve distinct types of assets, such as gated research, public blog posts, product documentation, and client portals. Implementing a hybrid ruleset allows companies to protect proprietary data without sacrificing discovery for customer acquisition pages.
Frequently Asked Questions
Does blocking grounding remove my site from Bing search results?
No. When webmasters set their AI training and grounding preferences, traditional Bing search ranking and indexing remain unaffected, provided the pages satisfy normal crawl and quality requirements. Grounding applies solely to AI-generated answers.
What is the difference between Disallow and DisallowExtended for grounding?
Setting grounding to Disallow prevents content from informing all generated responses across both first-party and third-party tools. DisallowExtended permits Microsoft-operated products like Copilot and Microsoft 365 to cite your material while blocking external platforms that access Microsoft grounding APIs, such as Web IQ or Foundry endpoints.
What happens if a website publishes no training or grounding tags?
Both preferences default to Allow when no applicable rule is detected. If you want Microsoft models to train on your content and use it for live grounding, no manual configuration is required.
Can page-level rules override robots.txt directives?
Yes, for training preferences. While AI training rules typically sit in robots.txt for a site or path, Microsoft allows page-level training directives to override the robots.txt rule for that specific URL.
Where This Fits in Real Projects
When Wasif Ahmed structures technical foundations for clients, governance around data ingestion is treated as a core architectural layer rather than an afterthought. Balancing discoverability against uncredited model ingestion requires precise robots.txt routing, custom CMS tag templates, and clean site delivery pipelines.
As global standards from bodies like the IETF take shape, business owners should review their web crawler policies systematically. Updating AI training and grounding preferences well ahead of the 2027 deadlines ensures search equity stays intact while controlling proprietary assets.
Managing AI training and grounding preferences carefully allows digital brands to maintain visibility in organic search while setting clear boundaries for model ingestion. If you want assistance updating your technical search architecture or automated workflows, reach out to Wasif directly.


