{"pageUrl":"https://promagen.com/why-your-site-is-invisible-to-chatgpt","lastModified":"2026-05-10","provenanceHash":"sha256:e5cfa4e9813a84f3eb5cdb41724a951d3d19f0f4f1d45fbded9e265456aa6b8c","provenanceNote":"OpenAI bot names and robots.txt behaviour reference platform.openai.com/docs/bots as of 2026-05-10. The sub-300ms TTFB threshold is operational guidance derived from Sentinel's measurement of when retrieval-driven AI engines reliably wait for response, not a vendor-published number. The five-reason fix order is operator-tested against the monitored site set; individual sites may have additional blockers not listed here.","claims":[{"id":"claim-five-structural-blockers","statement":"Five structural blockers account for the majority of \"ChatGPT cannot see our site\" diagnoses on commercial sites: (1) WAF / edge bot blocks, (2) robots.txt blocks of GPTBot or OAI-SearchBot, (3) slow server response (TTFB > ~300ms operationally), (4) missing or invalid JSON-LD schema, (5) orphan pages with no internal links. Each is a precondition for the next; fix order matters.","evidenceUrl":"https://promagen.com/sentinel/weekly","lastVerified":"2026-05-10","hash":"sha256:8e339e1ee907f8ecd7fc49b8e344e8388b8fd8bf944ec938c3054cbee8359362"},{"id":"claim-openai-autonomous-vs-user-triggered","statement":"OpenAI publicly documents GPTBot and OAI-SearchBot as autonomous crawlers controlled by robots.txt. ChatGPT-User is documented as a user-triggered fetcher; OpenAI's bot documentation calls out that robots.txt rules may not apply to user-initiated requests in the same way as to autonomous crawling. Blocking ChatGPT-User in robots.txt is not a reliable way to prevent user-initiated fetches.","evidenceUrl":"https://platform.openai.com/docs/bots","lastVerified":"2026-05-10","hash":"sha256:6b82dacb95f240769cf5be9e9109d517b6c911459fc09c0999596ae10c21c526"},{"id":"claim-edge-blocks-most-common-cause","statement":"WAF and edge firewall blocks are the most common cause of \"robots.txt looks fine but ChatGPT still cannot see us\" diagnoses. Many WAFs default-block traffic that looks like a bot regardless of the bot's legitimacy. The fix is to add explicit allow rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, and Bingbot in WAF / firewall configuration.","evidenceUrl":"https://promagen.com/sentinel/weekly","lastVerified":"2026-05-10","hash":"sha256:99cb09fd4d274c4a2ae732d6f1c6cf15c7196b6124deef049f68623ca85c9c6d"},{"id":"claim-300ms-ttfb-operational","statement":"Sub-300ms time-to-first-byte is the operational threshold at which retrieval-driven AI engines (ChatGPT Search, Perplexity) reliably wait for server response across query types. The 300ms number is not vendor-published; it is Sentinel's observation of where retrieval reliability drops off in practice.","evidenceUrl":"https://promagen.com/sentinel/weekly","lastVerified":"2026-05-10","hash":"sha256:024b6619421e6a10e73c09569ef1303e0efd9420bad69798356dc9b48a867dae"},{"id":"claim-orphan-pages-undiscoverable","statement":"A page with no internal links and no sitemap entry is undiscoverable by autonomous AI crawling. The engine cannot reach it unless a user pastes the URL via a user-triggered fetcher (ChatGPT-User). Internal linking is a precondition for autonomous citation eligibility.","evidenceUrl":"https://promagen.com/sentinel/weekly","lastVerified":"2026-05-10","hash":"sha256:d3160de43d9b27739b3349e39ebd6585f892580a0382d2ab376bee093ca25282"}]}