Forum Moderators: phranque
"For those of you who don't know, search engines use programs
known as spiders or crawlers to index content on Web sites.
Crawlers usually avoid pages with URLs that contain /cgi-bin or
bin (and other variants), as well as CGI escape characters like &,
?, =, etc. These pages often lead to massive databases with
recursive links that can easily trap crawlers in a maze of data.
Sometimes this poses a threat to the crawler, but more often
than not, a trapped spider will simply bring a server to its knees
in a couple of minutes. Although database creators usually don't
intend to trap search-engine crawlers, Bad People sometimes
create recursive crawlspaces for no other reason than pure
mischief. Generally, it's just a bad idea for a spider to follow
such links.
Unfortunately, there's not much you can do right now to get
the crawlers to index your site correctly. It may be tempting to
try to trick them into believing the pages are static HTML, but I
don't recommend this, since you still run the risk of endangering
your server if the crawler becomes trapped. "
I am not too happy about Google walking through all the dynamic pages the way they are. Consider the post you are reading right now has unlimited potential urls. There was one thread here on the system, they indexed at 10 different urls - that isn't right.
Fortunately, I remembered the thread about people's strange surfing habits, so I typed in this URL in the search box to click on the link to get here, like I had learned. Five results came up. I clicked on #5, and there was a sign-in page with my name on it.
