XML Sitemap Best Practices for Large and Small Websites

Include the preferred HTTPS URL for every page that should be indexed. The URL should return 200, allow crawling, permit indexing, and either self-canonicalize or clearly be the selected canonical. Treat inclusion as a quality assertion: this is a real page the site wants search engines to evaluate.

Exclude redirects, errors, noindex pages, duplicate parameters, internal search results, cart and account utilities, staging hosts, and alternate URLs that canonicalize elsewhere. A bloated sitemap sends conflicting signals and makes coverage reports harder to interpret.

A sitemap can contain up to 50,000 URLs and be up to 50 MB uncompressed, but operationally useful files are often smaller. Use a sitemap index to reference multiple files and group them by content type, locale, product family, or publishing system.

Segmentation creates observability. If blog URLs are indexed well but product sitemaps show a sudden error rate, the responsible template or release is easier to isolate. Keep each file on an allowed host and use absolute, consistently encoded URLs.

The lastmod value should reflect the last significant change to the page: an updated article, new product availability, revised documentation, or changed structured content. Do not set every URL to the time the sitemap was generated. A timestamp that changes daily for untouched pages teaches crawlers to ignore the signal.

Connect lastmod to the same publishing event that updates the page. Use full timestamps when time matters and dates when day-level precision is enough. Keep the value at or before the current time and consistent with visible dateModified markup on editorial pages.

Image and video sitemap extensions can help when media is a primary search asset and is difficult to discover in the page HTML. News sitemaps have their own freshness rules. For international sites, hreflang can be expressed in sitemaps, but it must remain bidirectional and consistent with page markup.

Do not add complexity simply because the schema permits it. A reliable basic sitemap is more valuable than a fragile sitemap generator with stale media metadata and inconsistent locale relationships.

Submit the sitemap index in search engine webmaster tools and compare submitted URLs with indexed outcomes. Differences are diagnostic clues, not automatic errors: a URL can be valid yet excluded because it is duplicate, low value, newly discovered, or contradicted by another signal.

Automate checks for invalid XML, non-200 entries, redirecting URLs, unexpected hosts, noindex conflicts, stale lastmod values, and sudden count changes. After a migration, compare the old and new inventories and keep redirects active while crawlers process the transition.

Large or frequently updated sites often generate sitemap files in a job and publish them to S3 behind CloudFront. Keep that pipeline deterministic: build from canonical production records, split files before protocol limits, write a sitemap index last, and make the release atomic enough that crawlers never see an index pointing at missing child files. Use stable object names when freshness is controlled by short cache lifetimes, or versioned names when the index can switch references safely.

Set content types correctly and verify that CloudFront does not cache error responses longer than successful sitemap responses. A scheduled Lambda or container task can regenerate the files, but it also needs alarms for failed runs, empty output, unexpected URL-count changes, and public fetch failures. The serverless file-processing architecture is a useful model for separating generation, storage, validation, and notification.

Sitemaps are discovery hints, not a substitute for internal links. Compare the generated URLs with the internal linking strategy guide, canonical tags, and HTTP status results. A URL that exists only in XML remains weakly integrated into the site and may signal that navigation or topic structure needs attention.