Tag
Crawling
9 articles, newest first.
SEO Nov 13, 2026
Google's status code page (last updated 4 February 2026) says 2xx content may be considered for indexing, redirects are followed up to 10 hops with 301 and 308 as strong signals, 4xx URLs…
3 min read
AEO Nov 10, 2026
Google says Search works in three stages: crawling, indexing and serving results, and not every page makes it through all three. It also says it does not accept payment to crawl a site more…
2 min read
SEO Nov 1, 2026
Google says its crawl budget guide is for very large sites, sites with 10,000 or more rapidly changing pages, or sites with many Discovered, currently not indexed URLs, and that the numbers…
3 min read
AEO Nov 1, 2026
Raw access logs show who asked for which page and what your server answered. Filter by user agent, verify claimed Google crawlers with a reverse then forward DNS check, count status codes…
3 min read
Google Oct 29, 2026
Google says a sitemap is a hint, not a guarantee. List the canonical, absolute URLs you want in search results, keep lastmod accurate, stay within 50 MB or 50,000 URLs per file, and submit…
3 min read
Google Oct 28, 2026
Google says a robots.txt file must be named robots.txt, sit at the root of the site host, be UTF-8 text, and contain groups that start with a user-agent line. Everything is crawlable unless…
2 min read
LLMs Aug 6, 2025
Recently, Cloudflare a global leader in web security and performance blocked the AI company Perplexity from accessing websites across its network. The reason? Cloudflare accused Perplexity…
3 min read
Google Aug 1, 2025
Googlebot has visited your page and fetched its content (crawling). But Google has decided NOT to include that page in its search index —at least for now. In short: Google knows about your…
1 min read
Google Oct 9, 2024
Ever wonder why Google doesn’t seem to be showing all your website pages in search results? Don’t worry, you’re not alone. This can be frustrating, but the good news is that there are some…
4 min read