lucy24 - 4:06 am on Jun 5, 2013 (gmt 0)
An ongoing head-scratcher:
A few months ago I made a /legal page because I was told-- probably by someone hereabouts-- that you're supposed to have one. Fine. Whatever makes the humans happy.
Q: What do robots want with this page?
I don't mean the ones that scoop up anything and everything that's linked directly from the front page. And I don't mean the search engines that home in on anything new. (In this case, they can crawl but not index.) I mean the ones that get the front page, the legal page-- and nothing else. Are they programmed to scan all links, and grab anything with the word "legal" or likely synonym? It's in a subdirectory, so they're not just guessing; besides, I never got 404s for this name before the page existed.
What are they looking for? Whatever it is, do I want them to find it?