The Hidden Wall: Why LLMs Can't See Your Catalog

The Audit: Simulating an AI Crawl

To test site readiness for AI Search (GEO), we ran a Python script simulating requests from major Large Language Models. The objective was to verify if ChatGPT or Perplexity could access the product catalog.

The result was a hard block. Every request returned a '403 Forbidden' status. The site was effectively firewall-blocked from AI indexing.

The Root Cause: WAF Configuration

The site utilized an enterprise Web Application Firewall (WAF) to mitigate scraping. The default rule set was overly aggressive, treating legitimate search crawlers from OpenAI and Anthropic as malicious botnets.

OpenAI OAI-SearchBot:   BLOCKED (403)
OpenAI ChatGPT-User:    BLOCKED (403)
Anthropic ClaudeBot:    BLOCKED (403)
PerplexityBot:          BLOCKED (403)

The Fix: Whitelisting via User-Agent

This was an infrastructure configuration issue, not a content issue. We updated WAF security rules to explicitly allow verified User-Agents and IP ranges from major search AI providers, while maintaining blocking rules for generic scrapers.

This update restored catalog visibility for the next generation of search engines.