Your Robots.txt Says No. Does Your AI-Built Site Want to Be Found?

You launched quickly. The landing page works, the signup form submits, and your first customer can finally visit the site.

Then someone asks, “Can Google find it?”

You check robots.txt and see this:

User-agent: *
Disallow: /

That single line tells compliant crawlers not to access anything on your website. Google may stop crawling your pages. Answer engines may miss your content. Your sitemap may sit untouched.

But here is the more important warning:

robots.txt does not protect anything.

It cannot secure .env files, database backups, admin routes, customer data, API keys, or private endpoints. It only gives crawlers instructions. It is not authentication, authorization, or a firewall.

For a website built quickly with AI or no-code tools, you need to check both sides:

• SEO: Can search engines discover and understand the public site?
• AEO: Can answer engines find, extract, and cite the right information?
• Security: Are private files and application routes actually protected?

Robots.txt is a signpost, not a locked door

A robots.txt file normally lives at:

https://yourdomain.com/robots.txt

It tells compliant crawlers which URLs they may access. Google explains that the file is mainly used to manage crawler traffic. It is not a method for keeping a page out of Google or protecting sensitive information.

Use authentication or password protection for private content. Use noindex when a publicly accessible page should not appear in search results.

What robots.txt can and cannot do

Situation	robots.txt	Real access control
Ask Google not to crawl a low-value URL	✅ Yes	Not usually needed
Tell OAI-SearchBot whether to crawl public content	✅ Yes	No
Help manage crawl volume	✅ Yes	No
Hide an API key	❌ No	✅ Yes
Protect an .env file	❌ No	✅ Yes
Secure an admin route	❌ No	✅ Yes
Protect customer records	❌ No	✅ Yes
Stop a malicious scraper	❌ Not reliably	✅ Authentication, authorization, WAF
Remove a page from search results	❌ Not by itself	✅ noindex, password protection, or removal

A crawler can ignore robots.txt. A curious person can read it. A security researcher can request a path directly. If the server returns a secret file, the problem is not your robots file. The problem is your server configuration.

The fastest way to break discoverability

This often happens during development:

User-agent: *
Disallow: /

The founder adds it to keep the unfinished site out of search. The website goes live. The rule stays in place.

The result:

❌ Google cannot properly crawl the homepage, product pages, documentation, or blog.

❌ OAI-SearchBot may not access public content for ChatGPT search results.

❌ PerplexityBot may not retrieve pages for citations.

❌ Your XML sitemap may exist but provide little benefit because the important URLs remain blocked.

❌ Your SEO score and AEO visibility can fall before you notice.

If your site is ready for customers, replace the temporary block with deliberate rules.

For a basic public marketing site, your file may be as simple as:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Do not copy this blindly. Review your actual site structure first. A public SaaS landing page and a customer dashboard do not need the same crawl policy.

Allow discovery without exposing private areas

You may want search engines and answer engines to access:

• Your homepage
• Product and service pages
• Public documentation
• Pricing
• Public articles
• Public case studies
• Contact information
• Public FAQs

You should not make private content public simply because you want the public site to be discoverable.

Use real controls for:

• /admin/
• /dashboard/
• /account/
• /api/private/
• /internal/
• /backups/
• .env and configuration files
• Database exports
• Debug pages
• Staging environments
• Customer records
• Uploaded files with private information

A robots file might reduce unnecessary crawling of a non-sensitive temporary area:

User-agent: *
Allow: /

Disallow: /tmp/
Disallow: /staging/

Sitemap: https://example.com/sitemap.xml

However, this does not secure /staging/. Put staging behind authentication, an IP restriction, or another server-level control. Remove sensitive files from public storage entirely.

Avoid advertising sensitive paths in robots.txt when possible. A line such as Disallow: /admin-secret-panel/ tells everyone that the path may be worth investigating.

Check search crawlers and answer-engine visibility separately

Search and training policies are not always the same.

OpenAI documents separate controls for OAI-SearchBot and GPTBot. OAI-SearchBot is used to surface websites in ChatGPT search. GPTBot relates to crawling content that may be used for training. You can allow one and disallow the other according to your policy.

If ChatGPT search visibility matters to your business, review the OpenAI crawler documentation.

Check these user agents in your file and server settings:

• Googlebot
• OAI-SearchBot
• PerplexityBot
• GPTBot
• Other crawlers relevant to your publishing policy

A cautious public-content setup could look like this:

User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This is only an example. Decide whether you want to permit training-related crawlers before adding blocks. Also check your hosting provider, CDN, firewall, and rate-limit rules. A crawler can be allowed in robots.txt and still be blocked by another layer.

✅ Allow public product documentation to be discovered.

❌ Block every crawler because you copied a development file from a template.

✅ Keep private customer routes protected by login and authorization.

❌ Assume a Disallow line prevents a visitor from opening the URL.

What llms.txt adds to AEO

An llms.txt file is a curated guide to important public content. It can help answer systems understand which pages matter and what each page covers.

It is useful for organizing:

• Product explanations
• Documentation
• Pricing
• Public FAQs
• Important guides
• Contact and company information

A simple example:

# Example Company

> Example Company provides bookkeeping software for small businesses.

## Product

- [Product overview](https://example.com/product): Main features and use cases.
- [Pricing](https://example.com/pricing): Current plans and included features.

## Resources

- [Small business bookkeeping guide](https://example.com/guides/bookkeeping): Practical guidance for business owners.
- [Frequently asked questions](https://example.com/faq): Answers about setup, billing, and support.

Keep it short, accurate, and public.

llms.txt is not a security control. It does not block a crawler. It does not grant access. It does not protect private content. It does not replace authentication, authorization, a firewall, or careful file permissions.

Use robots.txt to express crawl preferences. Use llms.txt to clarify your most useful public content. Use application security to protect everything private.

A five-step launch review

1. Read your robots.txt directly

Open:

https://yourdomain.com/robots.txt

Look for:

• Disallow: /
• Rules blocking the homepage
• Rules blocking CSS or JavaScript needed to render pages
• Missing sitemap location
• Rules copied from a staging environment
• Sensitive path names that reveal your application structure

Confirm that important public pages are crawlable.

2. Check your sitemap.xml

Open:

https://yourdomain.com/sitemap.xml

Confirm that it:

• Loads without an error
• Contains canonical public URLs
• Excludes login-only pages
• Excludes staging URLs
• Does not contain duplicate or broken links
• Matches the current website

Google’s robots.txt documentation explains the difference between crawl control and actual search removal.

3. Check public files and endpoints

Search your deployed project and public storage for:

• .env
• .env.local
• .git
• .zip
• .sql
• .bak
• Debug logs
• Configuration exports
• Test credentials
• API keys
• Private image or document URLs

Do not rely on robots.txt to hide these files. Delete them from public locations and rotate any credential that may have been exposed.

4. Test the customer-facing security basics

Review:

• HTTPS on every page
• Security headers
• Login rate limiting
• Password reset behavior
• Session expiration
• Cookie Secure and HttpOnly settings
• Appropriate SameSite settings
• Access checks on every private route
• Error messages that do not reveal secrets
• Dependencies that need updates

An AI-built site can look polished while still containing an unprotected route or outdated package. Test the actual deployed site, not only the code editor preview.

5. Check what crawlers can see

Review your hosting logs or crawler controls. Confirm that:

• Googlebot is not accidentally blocked
• OAI-SearchBot can reach intended public pages
• PerplexityBot is not blocked if citations matter to you
• Your firewall is not challenging legitimate crawlers unnecessarily
• Private routes still require authentication
• A user cannot change an ID in a URL to view another customer’s data

Then test after every major hosting, CMS, framework, or deployment change. Build tools can overwrite robots.txt without warning.

Before and after

Before	After
Disallow: / left over from development	Public pages intentionally allowed
Private route hidden only with robots.txt	Private route protected with authentication and authorization
Sitemap exists but lists blocked URLs	Sitemap contains current, canonical public URLs
Answer engines blocked accidentally	Public content reviewed for OAI-SearchBot and PerplexityBot visibility
llms.txt lists every possible URL	llms.txt curates a small set of useful public pages
.env file sits in public hosting	Secrets removed, credentials rotated, deployment checked
Login form accepts unlimited attempts	Rate limiting and secure session settings enabled
Dependencies never reviewed	Packages updated and tested regularly

Fix it without rebuilding the product

You do not need to throw away a working site.

Start with the smallest safe changes:

1. Remove an accidental Disallow: /.
2. Generate a clean robots.txt file.
3. Submit or verify your sitemap.
4. Add or update public answer-focused pages.
5. Create a short, accurate llms.txt file.
6. Protect private routes with real access controls.
7. Remove exposed files and rotate credentials.
8. Update dependencies.
9. Re-test the live website.

The Smart Analytics Free Tools page includes a Robots.txt Generator for creating a clean copy-paste file and an llms.txt Generator for organizing public content for answer engine discovery.

You can also use the Schema Markup Generator to create structured data and the Meta Tag Generator to improve how pages appear in search previews. The Search Console Regex Library provides copy-paste patterns for finding crawl, indexing, and query problems.

Everything on the Free Tools page is free. There is no login, no email requirement, and no paywall.

Final check before you send customers to the site

A discoverable website is not automatically a secure website.

✅ Let compliant crawlers reach the public pages you want found.

✅ Help answer engines understand your products, services, and expertise.

✅ Use a sitemap and clear page structure.

✅ Protect private routes with authentication and authorization.

✅ Remove secrets from public files.

❌ Do not use robots.txt as a password.

❌ Do not assume llms.txt controls access.

❌ Do not leave development blocks in a live deployment.

Near the end of your review, run the free Smart Analytics SEO and AEO analyzer to check your SEO score, AEO score, robots.txt, sitemap, structured data, page speed, and other ranking factors. No login, email, or paywall is required. You receive the full report.

Run the separate Vulnerability Check to look for exposed files, leaked credentials, unsafe login forms, missing protections, cookie issues, and outdated software. The security score is shown on screen, while specific findings remain private in the downloadable PDF.

Your site launched quickly. Take one more pass before assuming it is ready.