How Google Crawls and Indexes Your Website

Before a page can rank, Google must find it, read it, and store it. These steps are called crawling, rendering, and indexing. Understanding how they work makes it much easier to diagnose why a page is missing from search results.

Step 1: Discovery

Google finds pages mainly by following links from pages it already knows, and by reading your XML sitemap. A page with no links pointing to it and no sitemap entry may never be found. This is why internal linking matters.

Step 2: Crawling

Googlebot, Google's crawler, requests the page from your server. Whether it can do this depends on:

Your robots.txt, which can block certain paths
Your server responding properly and reasonably quickly
The page not requiring a login

Google decides how often to visit based on factors such as how important and how frequently updated your site seems, and how well your server copes. Small sites are generally crawled fine, but slow or error-prone servers can limit it.

Step 3: Rendering

Google processes the page much as a browser does, running JavaScript to see the final content. If key text or links appear only after complex scripts, they may be seen later or missed. Using straightforward HTML for important content is safer.

Step 4: Indexing

Google analyzes what the page is about and decides whether to store it in its index. Pages may not be indexed if they:

Have a noindex instruction. See noindex and nofollow
Are duplicates of another page. See duplicate content
Are very thin or low quality
Return errors

Crawled does not always mean indexed. Indexing is selective.

Step 5: Ranking

When someone searches, Google picks relevant pages from its index and ranks them using many signals, including relevance, quality, usability, and links. Appearing in the index is necessary but not sufficient to rank.

How to see what Google sees

Search Console is the essential tool:

The page indexing report lists indexed and non-indexed pages with reasons
The URL Inspection tool shows whether a specific page is indexed, the canonical Google chose, and how it was rendered
Crawl stats show how often Googlebot visits and any server problems
A site search for site:yourdomain.com gives a rough view, but Search Console is more reliable

How to help Google

Link important pages from your navigation and content
Keep a clean, accurate sitemap
Make pages fast and error-free. See how to reduce load time
Avoid accidental blocks and noindex tags
Use clear titles and helpful content. See on-page SEO checklist
Remove or redirect dead pages. See 301 redirects

If you launched a new site, see how to get a new website indexed.

The bottom line

Pages must be discovered, crawled, rendered, and chosen for the index before they can rank. Make that path smooth, and use Search Console to see where it breaks.