New independent field guide, published 17 October 2026. Examples are illustrative. The historical directory records linked below are separate source context, not current recommendations or earlier versions of this essay.
A page can be reachable by a browser, permitted for a crawler, eligible for indexing and selected by an editor. Those conditions are related only in limited ways. Confusing them leads to bad publishing decisions: a robots file is treated as security, a sitemap as a ranking promise or a directory listing as proof that a resource has been checked. This guide separates the technical and editorial layers using invented examples. It explains the basic purpose of crawler instructions, then develops a practical publishing and review model. It does not offer traffic guarantees or claim that the historical directory's old search-related wording establishes present search performance.
Start with the request boundary
A publicly reachable address is one for which a requester can obtain some response under particular conditions. That response may be content, a redirect, an access challenge or an error. Reachability is not the same as permission to use every part of a service, nor is it proof that the returned material matches a directory description. Keep the request observation narrow. In a fictional collection, a resource may load in a browser while rejecting an automated tool. That establishes a difference in observed access paths, not a universal verdict about its quality or availability.
Actual access control is enforced through the service's authentication and authorisation mechanisms, not through an editor's wish that a page remain obscure. A resource that must not be public needs appropriate access restrictions. Publishing its address in a less prominent place does not create a security boundary. This distinction matters for directories because they organise destinations they may not control. A directory can decide what to link and how to describe it, but cannot make a third-party document private or safe merely by assigning it to a particular category or hiding its card from casual browsing.
Robots instructions address crawler behaviour
RFC 9309 specifies the Robots Exclusion Protocol. Its rules describe behaviour crawlers are requested to honour, and the document explicitly distinguishes those rules from access authorisation. A robots file can contain groups for crawler identifiers and path-related allow or disallow instructions. That is a useful coordination mechanism, but it is not an authentication system. Do not place confidential information at a public address and assume a disallow rule protects it. The file can communicate a crawling preference while leaving the underlying request technically possible for another client.
An invented publisher might disallow a set of duplicate search-result paths to reduce unwanted crawling. That choice does not mean the pages are necessarily low quality, dangerous or absent. It says something specific about the publisher's crawler instructions. Conversely, allowing a path is not a recommendation or a copyright licence for every use of its content. For directory work, record robots-related observations only when relevant to the task and interpret them at that layer. Do not convert a technical instruction into a broader editorial statement about whether the linked resource deserves inclusion.
Indexing is not the same operation as crawling
Crawling concerns fetching resources; indexing concerns how a search system stores and represents information about them. A publisher can provide indexing instructions in supported page metadata or response headers, but a crawler needs a way to encounter the relevant instruction. Blocking retrieval and asking a system to inspect a page-level instruction can therefore work at cross purposes. The exact handling depends on the search system and supported controls. A robust publishing plan identifies the intended outcome and verifies it through appropriate evidence rather than assuming that one file simultaneously manages every aspect of search visibility.
For an invented archive edition, new explanatory articles can be intended for discovery while historical source records are marked noindex and omitted from the sitemap. That expresses an editorial separation between new writing and preserved context. It does not make the archive private, and it does not guarantee that every external system will immediately remove or ignore every reference. The archive can remain openly linked for readers who want provenance. State the intended role clearly: discoverable new explanation and accessible historical evidence are different kinds of page, not mutually exclusive definitions of whether a site exists.
A sitemap is an inventory aid, not a ranking promise
A sitemap can help communicate selected URLs, but its presence does not establish that all listed pages will be indexed or ranked in a particular way. Likewise, a page omitted from a sitemap may still be discoverable through links. Treat the file as one part of a coherent publishing structure, alongside navigation, canonical addresses and page-level metadata. The list should reflect the site's intended public collection rather than becoming a dumping ground for every possible query variant. An inventory is useful when its entries have a clear identity and purpose.
Imagine a fictional directory with thousands of near-duplicate search combinations. Adding every combination to an inventory does not create thousands of useful editorial resources. The publisher should decide which pages provide distinct value and maintain stable destinations for them. This is an information-design question before it becomes a technical configuration question. A clear category page with good descriptions may serve readers better than a sprawling set of barely different result pages. No particular number of indexed URLs is inherently a success metric without considering whether those URLs help people find the material the collection was built to organise.
Editorial inclusion has its own evidence
A resource can satisfy an editor's relevance criteria regardless of whether a particular crawler currently indexes it. Conversely, a highly visible search result may fail the directory's scope or review policy. Keep the editorial decision separate from technical visibility. In an invented educational collection, the editor might include a specialist public document because it answers a defined question, then note an access limitation if one was observed. Search prominence is not a substitute for inspecting the resource and explaining why it is useful for the intended audience.
Old search-related marketing language deserves especially careful treatment in an archive. A historical claim that a directory is SEO friendly records what the old site said; it does not establish present ranking effects or a current service promise. This edition preserves its authentic web-directory subject while excluding obsolete ranking badges and active paid submission flows. New guides discuss structure, maintenance and evidence rather than promising traffic gains. Historical relevance and present endorsement are different claims, just as crawl permission and editorial selection are different processes. A label should make the relevant process visible rather than blending them together.
Verify the layer you intended to change
When changing a publishing control, write down the expected observable result. A robots-rule change calls for inspecting the served file and applicable matching behaviour. A page-level indexing instruction calls for checking the actual response or HTML. A navigation change calls for testing the reader's route. A directory review calls for inspecting content and scope. None of those checks can stand in for all the others. Keeping the boundary explicit prevents a successful technical response from being reported as a completed editorial or search-visibility outcome that has not actually been observed.
A practical checklist can remain short: identify the resource, state the desired access and discovery roles, configure the relevant controls, inspect the delivered result and record any unresolved external behaviour. The archive links below preserve old directory categories and homepage context; they are not evidence of current indexing or quality. The lasting distinction is between what a client can request, what a crawler is instructed to do, what an index may represent and what an editor has chosen to include. A useful publishing workflow respects those boundaries instead of asking one green indicator to certify the entire chain.
References & further reading
- IETF — RFC 9309: Robots Exclusion Protocol ↗
- Google Search Central — Robots meta tag and X-Robots-Tag specifications ↗
- Google Search Central — Learn about sitemaps ↗
From the recovered archive
- Directory homepage — 2010-11-17 · captured 17 November 2010
- Computers & Internet / Web Directories · captured 25 December 2010