# As a condition of accessing this website, you agree to # abide by the following content signals: # (a) If a content-signal = yes, you may collect content # for the corresponding use. # (b) If a content-signal = no, you may not collect content # for the corresponding use. # (c) If the website operator does not include a content # signal for a corresponding use, the website operator # neither grants nor restricts permission via content signal # with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search # results (e.g., returning hyperlinks and short excerpts # from your website's contents). Search does not include # providing AI-generated search summaries. # ai-input: inputting content into one or more AI models # (e.g., retrieval augmented generation, grounding, or other # real-time taking of content for generative AI search # answers). # ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS # RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN # UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS # IN THE DIGITAL SINGLE MARKET. # Aurora — robots.txt # # Training and search are SEPARATE controls at every vendor. They are granted # separately below, on purpose: # # search bots — allowed without qualification. Aurora cannot be recommended # by an assistant that is not permitted to read the site. # training bots — allowed, for everything this origin serves. There is no # Disallow anywhere below, deliberately: every document here # is prose Aurora published on purpose, and no GET path on # this host returns user data. Refusing training would buy # no user privacy. What may be done with the text afterwards # is a separate question, answered by the content signals # below and recorded in docs/geo/README.md. # # WARNING: robots.txt is not the only gate. A CDN can block AI crawlers before # the request reaches this file — Cloudflare has done so by default on new # domains since 2025-07-01. Verify with a real request per user-agent, and check # the origin logs, before trusting anything in this file. # # Changes here take roughly 24 hours to propagate to OpenAI's systems. # ── Content signals ────────────────────────────────────────────────── # Access and usage are different questions, and the notice at the top of this # file answers the second one. The Content-Signal line repeats in every group # below because a crawler obeys only the single group that matches it best: a # signal stated once under `User-agent: *` would never be read by GPTBot or # ClaudeBot, which have groups of their own here. # # All three values are `yes`, and not one of them is an oversight: # # search=yes — the access grant above, restated as a usage right. Aurora # cannot be recommended by an assistant that may not index # it. # ai-input=yes — grounding and retrieval at answer time. This is the value # that decides whether an assistant may read this site while # it answers someone asking for a private meeting recorder. # Refusing it would withhold Aurora from the one question it # exists to answer. # ai-train=yes — the decision recorded in docs/geo/README.md: this origin # serves only prose Aurora published on purpose, and no GET # path here returns user data, so refusing training would # buy no user any privacy. # # Nothing is refused, so the reservation of rights in the notice above reserves # nothing. That is the intended reading: this file grants, it does not withhold. # ── OpenAI ─────────────────────────────────────────────────────────── # Search: governs whether Aurora can appear in ChatGPT search answers. User-agent: OAI-SearchBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # User-directed: a person asked ChatGPT to open a page. User-agent: ChatGPT-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # Training. User-agent: GPTBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # ── Anthropic ──────────────────────────────────────────────────────── User-agent: Claude-SearchBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / User-agent: Claude-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / User-agent: ClaudeBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # ── Google ─────────────────────────────────────────────────────────── # Googlebot also feeds AI Overviews and AI Mode. User-agent: Googlebot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # Google-Extended is a control token with no user-agent of its own. It governs # Gemini training AND grounding — the content Gemini is given at prompt time. # Disallowing it removes Aurora from Gemini's grounding without any gain. User-agent: Google-Extended Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # ── Microsoft / Copilot ────────────────────────────────────────────── User-agent: bingbot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # ── Perplexity ─────────────────────────────────────────────────────── User-agent: PerplexityBot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # Perplexity-User is user-initiated and generally ignores robots.txt anyway. User-agent: Perplexity-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # ── Others ─────────────────────────────────────────────────────────── User-agent: Applebot Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / User-agent: Applebot-Extended Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / User-agent: cohere-ai Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / User-agent: MistralAI-User Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / # xAI publishes no crawler documentation and Grok's retrieval has been reported # using rotating residential IPs and spoofed browser user-agents. There is no # token here that would reliably reach it. The defense is not a robots rule: it # is that every page works for an anonymous, JavaScript-free, single-shot GET. # ── Everything else ────────────────────────────────────────────────── User-agent: * Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Sitemap: https://auroraisyours.com/sitemap.xml