# Bostad.io — crawler policy. This host serves the frontend: the public
# marketing pages (prerendered, open) plus the sign-in app served as a
# client-only shell (/workspace, /login, /logout) — the app is not meant to be
# indexed. The API lives on api.bostad.io, which carries its own robots.txt.
#
# robots.txt is advisory, not access control: list only paths that exist on
# this host and merely shouldn't be indexed. Anything sensitive is protected
# by auth, not by omission from this file.
#
# Two groups with the same rules today; the split records a decision, not a
# difference (#508, 2026-09-25). Group 1 is search and user-initiated fetches,
# which put pages in answers with a link back. Group 2 is model training, allowed
# on purpose. If a rule changes, change it in both groups.

User-agent: *
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: meta-webindexer
User-agent: meta-externalfetcher
Allow: /
Disallow: /workspace
Disallow: /workspace/
Disallow: /login
Disallow: /logout
Disallow: /reset-password
Disallow: /invite
Disallow: /forgot-password

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Allow: /
Disallow: /workspace
Disallow: /workspace/
Disallow: /login
Disallow: /logout
Disallow: /reset-password
Disallow: /invite
Disallow: /forgot-password

Sitemap: https://bostad.io/sitemap.xml