Skip to main content

SEO, robots.txt, sitemap, and Basic Analytics

When building a first site, SEO is often misunderstood in two extreme ways:

  • Either you think you can ignore it completely
  • Or you think it is a complex marketing project to handle later

For personal sites and documentation sites, what matters more is getting the basics right first.

What Basic SEO Actually Is​

It is not "gaming the system." It is making sure search engines can at least:

  • Find your pages
  • Crawl your content
  • Understand the page topic
  • Not be blocked by misconfigurations

Start With These Fundamentals​

1. Clear Page Titles and Descriptions​

Do not have vague titles across the entire site. Documentation pages should accurately express their topic.

Do not use one set of paths today and a different set tomorrow, with no migration strategy.

Main articles should be reachable from each other. Do not let content become isolated islands.

What Is robots.txt For​

It is a convention file that tells crawlers "which paths can be crawled and which are not recommended."

It is not a permission system and not a security boundary.

What Is sitemap For​

Think of it as "a page list for search engines."

It is helpful for content sites because it more explicitly tells search engines:

  • Which pages exist on the site
  • Which pages are worth discovering

Why Basic Analytics Also Matters​

You should at least know:

  • Whether anyone is visiting the site
  • Which pages are entry points
  • Which pages have abnormal traffic
  • Whether there are obvious 404s or crawling issues

Most Common Pitfalls​

1. robots.txt Misconfigured to Block Yourself​

This is worse than "not doing SEO" because it directly affects crawling.

2. All Page Titles Are Duplicated​

This makes it hard to distinguish content in search results and when sharing.

3. Messy Content Structure With No Entry Pages​

Search engines, like readers, prefer sites with clear structure.

4. Changing Paths Without a Migration Plan​

Old links and index results will be affected.

Most Practical Advice For Personal Sites​

  1. Make sure page titles are accurate
  2. Make sure main content is directly accessible
  3. Make sure documents have reasonable internal links
  4. Make sure robots.txt and sitemap have no low-level errors
  5. Make sure the site has at least basic access analytics