Skip to content
Back to Launch & Discoverability

Indexability Check

Verifying that search engines can actually crawl and index your new pages — robots.txt, meta robots, canonical, and sitemap inclusion all aligned.

Why

Indexability is the gate before ranking: if Google can't index your page, it will never rank, no matter how good the content is. The most common launch SEO failure is a page that is technically live, linked from the homepage, but de facto noindexed by a misconfigured robots.txt, a canonical pointing to another URL, or a meta robots noindex tag left over from staging. These issues are invisible to users (the page renders fine) and invisible to basic monitoring (the page returns 200), so they persist for months until someone notices traffic isn't growing. An indexability check verifies the full chain: robots.txt allows the path, the page has no noindex tag, the canonical is self-referencing, the URL is in the sitemap, and Google's URL Inspection tool confirms 'indexable'. This 5-minute check per page prevents the 2-month penalty of an unindexable page.

When

After deploy, within 24 hours of launch. Use Google Search Console's URL Inspection tool on 5-10 key new URLs to confirm indexable status.

How

1. Check robots.txt: ensure it does not Disallow the new paths (e.g. /launch, /launch/*); common mistake is a broad Disallow like /*.astro that accidentally matches /launch
2. Check meta robots: view source on 5 new pages and confirm there is no <meta name="robots" content="noindex"> tag; if there is, remove it or change to "index,follow"
3. Check canonical: each new page's <link rel="canonical"> must point to its own URL (self-referencing), not to the homepage or a different page; a canonical to another URL tells Google 'don't index me, index that instead'
4. Check sitemap inclusion: every new URL must appear in sitemap-pages.xml; fetch the sitemap in the browser and Ctrl+F for the new paths
5. Submit the sitemap to Google Search Console (Crawl > Sitemaps > Add sitemap-pages.xml) and Bing Webmaster Tools
6. Use Search Console URL Inspection: paste 5-10 key new URLs, click 'Request Indexing'; the tool tells you if the URL is indexable, and if not, why (blocked by robots.txt, canonical, noindex, etc.)
7. Check server response: curl -I https://ez4code.com/launch must return 200 (not 301, 404, or 500); a 301 to another URL means the canonical chain is broken
8. Check render: use Google's Rich Results Test or Search Console's Live Inspection to confirm Google can render the page (JS-heavy pages sometimes render blank to crawlers)
9. Monitor for 2 weeks: check Search Console 'Coverage' report weekly; new pages should move from 'Discovered' to 'Indexed' within 2-4 weeks

Example

# Indexability Check — ez4code v2.0 /launch

## 1. robots.txt
User-agent: *
Disallow: /astro/
Disallow: /api/
# /launch NOT disallowed ✓

## 2. Meta robots (view source /launch)
# No <meta name="robots" content="noindex"> found ✓

## 3. Canonical
<link rel="canonical" href="https://ez4code.com/launch">
# Self-referencing ✓

## 4. Sitemap inclusion
# sitemap-pages.xml contains:
<url><loc>https://ez4code.com/launch</loc></url>
<url><loc>https://ez4code.com/launch/pre-launch-checklist</loc></url>
# All 30 new /launch/* URLs present ✓

## 5. Sitemap submitted
Google Search Console: sitemap-pages.xml → Submitted ✓
Bing Webmaster Tools: sitemap-pages.xml → Submitted ✓

## 6. URL Inspection (Search Console)
/launch                    → "URL is on Google" ✓
/launch/pre-launch-checklist → "Indexable" ✓
/launch/ph-preparation     → "Indexable" ✓
/launch/seo-launch-audit   → "Indexable" ✓
/launch/geo-targeting      → "Indexable" ✓

## 7. Server response
curl -I https://ez4code.com/launch → HTTP/2 200 ✓

## 8. Render (Rich Results Test)
/launch renders fully (all content visible to Googlebot) ✓

## 9. 2-week monitor
Week 1: 18/30 pages Indexed, 12 Discovered
Week 2: 28/30 pages Indexed, 2 Discovered
=> Indexability confirmed

Common Problems

  • ⚠A broad robots.txt Disallow that accidentally blocks new paths — 'Disallow: /*.astro' is meant for build assets but also matches /launch if the path contains 'astro'; test robots.txt with Google's tester before launch.
  • ⚠Canonical pointing to the homepage — a CMS default or a copy-paste error sets every page's canonical to '/', telling Google to index only the homepage and deindex everything else; this is a site-killing mistake.
  • ⚠JS rendering that hides content from Googlebot — a React island that renders content only on client hydration shows blank to the crawler; use SSR (Astro's default) or prerender so the content is in the HTML.
  • ⚠Sitemap not updated — the sitemap still lists the old URLs and not the new /launch pages; Google crawls the sitemap but never discovers the new pages, delaying indexing by weeks.

Best Practices

  • ✓Use Google's robots.txt Tester (Search Console > Crawl > robots.txt Tester) before launch — it shows exactly which paths are allowed/blocked, catching accidental over-blocks in 30 seconds.
  • ✓Set canonical tags via the framework (Astro's <link rel=canonical> in the layout), not manually per page — a single source of canonical logic prevents per-page copy-paste errors.
  • ✓Prerender all content pages (Astro's default for .astro pages) so the HTML contains the full content, not a JS hydration stub — Googlebot can index prerendered HTML immediately but waits weeks for JS-rendered content.
  • ✓Regenerate and resubmit the sitemap on every deploy, not just at launch — a stale sitemap means new pages (added in v2.1) are never discovered; automate sitemap generation in the build.