Lina Irawan
For two decades, the discipline of Search Engine Optimization (SEO) was built on a foundation of "portable guidance." It was a golden age of predictability: if Google released a new set of guidelines regarding sitemaps, canonical tags, or structured data, practitioners could rest assured that those efforts would be respected by Bing, Yahoo, and the broader search ecosystem. This wasn’t merely a stroke of good fortune; it was the result of a deliberate, two-decade-long effort by major search engines to build a shared, interoperable infrastructure.
However, that era of collaborative standards has come to an abrupt end. We have entered the age of Large Language Models (LLMs), an environment where search behavior is dictated by proprietary architectures, divergent training corpora, and distinct alignment processes. The "old guard" habit of treating one engine’s guidance as a universal map is no longer just ineffective—it is a dangerous blind spot that threatens to leave digital marketers invisible to the majority of the AI-driven web.
The Shared Standards That Built the SEO Era
To understand why the current landscape feels so fractured, one must look at the structural collaboration that defined the early 2000s. The portability of SEO guidance was not an accident of technology; it was a policy choice.
In November 2006, Google, Yahoo, and Microsoft (Bing) formally agreed to support a common sitemap protocol (version 0.90), building upon Google’s earlier foundation. This was a watershed moment. It signaled that these fierce competitors understood that a common vocabulary for webmasters was mutually beneficial. This sentiment reached its zenith on June 2, 2011, with the launch of Schema.org. By establishing a shared vocabulary for structured data, the engines made it possible for webmasters to invest effort once and see that investment pay dividends across the entire search spectrum.
This spirit of cooperation extended to the very infrastructure of the web. The robots.txt convention, established in 1994, was formalized as RFC 9309 at the IETF in 2022, ensuring that every serious crawler adhered to the same rules of engagement. Even more recently, the IndexNow protocol—launched by Microsoft Bing and Yandex in 2021—attempted to continue this tradition of shared efficiency. While Google opted not to fully adopt IndexNow, the precedent was clear: the major search players historically prioritized a shared "substrate" that made the web more navigable for everyone.
The Great Divergence: Why LLM Stacks Don’t Align
In the LLM landscape, the shared substrate has vanished. The differences between providers like OpenAI, Anthropic, Google, and Perplexity are not merely cosmetic; they are baked into the core of how these models function.
1. Training Data and Licensing
The foundational knowledge of these models is radically different. OpenAI has aggressively secured licensing deals with massive publishers like News Corp, Axel Springer, Reddit, and The Atlantic. Conversely, Anthropic and other providers have pursued different, often undisclosed, strategies for acquiring data. Because these models are "fed" by different documents, they arrive at different conclusions. A brand cannot assume that its presence in a Google-indexed corpus ensures its inclusion in a Claude-driven response.
2. Crawler Infrastructure
The days of a single, omnipotent "bot" are over. Today, providers operate multiple, specialized agents. OpenAI utilizes GPTBot for training, OAI-SearchBot for indexing, and ChatGPT-User for retrieval. Anthropic maintains ClaudeBot, Claude-SearchBot, and Claude-User. Google has introduced Google-Extended specifically to manage training access for Gemini, separate from its traditional Googlebot. These bots serve different masters and obey different instructions, meaning a single robots.txt directive is no longer a "one-size-fits-all" solution.
3. Retrieval and Alignment
The retrieval architectures themselves are fundamentally disconnected. ChatGPT relies heavily on the Bing index, while Perplexity utilizes a sophisticated Vespa-based pipeline. Gemini grounds its responses in the Google Knowledge Graph. Even when the input is the same, the "alignment layer"—the post-training process that dictates tone, safety, and formatting—ensures that the final output is unique to that specific model. Whether through Reinforcement Learning from Human Feedback (RLHF) or Constitutional AI, these models are being trained to "think" in ways that are intentionally idiosyncratic.
When Guidance Fails: The llms.txt Case Study
Perhaps the most poignant example of the current disconnect is the rise and fall of the llms.txt file. Proposed in late 2024 by Jeremy Howard of Answer.AI, the file was envisioned as a markdown-based manifest to help LLMs identify the most relevant content on a site. It was hailed as the "new sitemap" and quickly integrated into agency service catalogs and SEO tools.
Yet, as of mid-2026, no major LLM provider has officially adopted it. Server-log analysis reveals that major AI crawlers rarely request the file. Google’s John Mueller has even compared the concept to the deprecated and largely useless "meta keywords" tag. This failure illustrates the fundamental shift: the SEO community tried to impose a standard from the outside, but the platform providers—who are currently in a high-stakes race to differentiate their products—have no interest in adopting a universal standard.
The Gemini Inversion: A House Divided
The most jarring evidence of this fragmentation exists within Google itself. For two decades, Google’s "Search Central" documentation was the bible of SEO. Today, however, that documentation is becoming increasingly decoupled from the reality of Google’s own AI products.
Data from Ahrefs and BrightEdge shows that the overlap between pages ranking in the traditional "Top 10" and those cited in AI Overviews has plummeted. When Google upgraded to the Gemini 3 model in early 2026, the citation patterns shifted drastically, with newer models favoring different sources than their predecessors.
The divergence is even more stark when comparing Google’s traditional search results to its newer "AI Mode." Semrush data indicates that while these surfaces often reach similar semantic conclusions, they cite the same URLs less than 14% of the time. This is the "Gemini Inversion": following Google’s official SEO documentation may secure a high ranking in traditional search, but it provides no guarantee of visibility in the AI-powered interfaces that are rapidly claiming the "zero-click" real estate.
Implications for the Future of Search
The practical implications for digital marketers are profound. The old reflex—optimize for the dominant player and trust in the trickle-down effect—is now a liability.
1. The Death of Universal Optimization
Practitioners must stop treating any single LLM provider’s guidance as a universal map. If Google releases a guide to "Gemini Optimization," that information is specific to the Gemini ecosystem. It does not provide insight into how Perplexity or Claude perceive your content.
2. The Rise of Platform-Specific Audits
Testing must become platform-specific. Brands should monitor their visibility not just in Google’s traditional SERPs, but across ChatGPT, Perplexity, and Claude. A brand may be an authority in one ecosystem and completely invisible in another.
3. A Focus on the "Universal Layer"
While the shared substrate has shrunk, it has not disappeared. Crawler accessibility remains paramount. High-authority, factual, primary-source content—the kind that gets cited by Wikipedia, Reddit, and major news outlets—still acts as a force multiplier. Earning a reputation on these "trust hubs" is the only remaining strategy that carries over into nearly every LLM retrieval system.
Conclusion: A New Era of Complexity
The SEO industry is currently being rebuilt. We are moving from an era of "shared protocols" to an era of "platform-specific integration." The convenience of the last twenty years, where one set of rules governed the entire search landscape, has been replaced by a fragmented, highly competitive, and volatile environment.
The reality is that we now have more work to do with fewer guarantees. Success in the age of LLMs requires a departure from the "one-size-fits-all" mentality. Practitioners who recognize this divergence early, who build distinct strategies for different AI architectures, and who stop relying on the illusion of universal search, will be the ones to define the standards of the next decade. The era of the "universal map" is over; it is time to start charting the individual territories.
Lina Hope
Rifan Muazin
