Atlas crawls websites you already publish — docs, blogs, knowledge bases — and lets your AI agent search and read them. It is not a CMS; for content you write yourself, use the Library.

Atlas is your team's shared map of external websites (docs sites, blogs, knowledge bases) that your AI agent can search and read pages from. Add a root URL, Atlas finds the sitemap (or crawls if there isn't one), charts each page, and stores a clean Markdown rendering for retrieval.

Atlas vs the Library

People sometimes assume Atlas is a place to write content. It is not. Use the right tool for the job:

Library Atlas
What it stores Articles you author in cStar Pages crawled from external websites
Where the content lives cStar database The external site (Atlas just indexes)
Editor Yes, rich-text editor in the dashboard None. Content updates when you re-chart
Public-facing Yes (your library URL, embeddable widgets) No, agent-only
Best for Help center articles, FAQs, support docs you write Existing docs, marketing sites, third-party knowledge bases the agent should reference
Updates When you save When you re-chart (manually or scheduled)

Use both: write your support content in the Library (it's also the content powering Quick Help and the public help center), and point Atlas at any external URL you want the agent to be able to quote, like your product docs site, your changelog, or your blog.

How it works

  1. You give Atlas a root URL (e.g. https://docs.acme.com).
  2. Atlas runs sitemap discovery: tries robots.txt, then sitemap.xml, then sitemap_index.xml, then a crawl fallback if none of those exist.
  3. For each discovered URL, Atlas fetches the page, extracts the main article content (drops nav/footer/sidebars), and stores Markdown + a search excerpt.
  4. When the agent answers a question, it can search Atlas alongside your Library, then read full pages when it needs deeper context.
  5. You re-chart (manually or on a schedule) when the source site changes.

Where it lives in the dashboard

Atlas is configured per agent because it's a per-agent ability:

Settings → Agent → Abilities → Atlas access

Toggle Atlas access ON and you'll see the team's shared sources. Add a source (team-wide, all agents see it), then use the per-row toggle to pick which sources this agent can read. A source can be enabled for some agents and disabled for others. Useful if you have, say, an internal-facing agent that should see both your customer docs and your internal runbook, while the external agent should only see customer docs.

What gets crawled

Atlas tries to be a polite citizen of the websites you point it at:

  • Honors robots.txt
  • Skips common low-signal patterns by default: WP archive pages, query-heavy URLs, RSS feeds, admin paths
  • Respects noindex meta tags
  • Caches with reasonable headers; doesn't hammer the source

You can override the auto-skip rules per source if you want broader coverage.

Limits

  • 20 sources per team
  • Each source can have thousands of pages (no hard per-source page cap, but huge sites benefit from exclude_patterns)
  • Re-chart respects the same politeness rules as the initial chart

Next steps