How Google May (Theoretically) Discover Web Pages

Very often you find Google knows much more about your site than it is supposed to: you find it crawl pages with 0 backlinks (you are aware of) or you can find pages that have never really existed in its index.

This brings us to a plenty of speculations, theories and observations as to what can be used for a web page discovery; here are 15 of them (article inspired by WebmasterWorld thread):

1. “Dofollow” “direct links (either external or internal) links pointing to a page;
2. ‘URL manipulation‘ – i.e. if site.com/?one-word exists, then perhaps so does site.com/?two-words.
3. Link inside the forms:

Matt [Cutts] confirmed that such “links” send PageRank. Google establishes a virtual link on their back end when they find something through form navigation and they add that virtual link to the webgraph.

4. Clicking a link using a browser with Google toolbar installed (or a pagerank indicator of some sort that sends every page you visit to Google);
5. Putting the link in Google searchbar and performing the search for it (you may be surprised to know how many people use Google to navigate the web instead of regular browser address bar);
6. Other sites hotlinking to your images;
7. Other sites linking to your javascript or CSS files;
8. Links in email a search engine has access to (link in Gmail);
9. URLs within meta data of graphics and video files;
10. URLs within HTML comments; URLs within the head section, meta data of an HTML page, or alternate html entities (alt, name, id, etc) or any other HTML attributes;
11. Links in Flash movies (games, quizzes, etc);
12. Non-linked URLs (http://www.domain.com);
13. Links in any documents other than web pages e.g. .doc, .pdf, .txt, etc – see detailed experiment on Search engines and pdf ;
14. Links in other Google produced software (gadgets, widgets)
15. Advertising links (AdWords/Yahoo), and other services like Maps.

Let’s watch this list grow with your ones!