Welcome to WebmasterWorld Guest from

Forum Moderators: phranque

Message Too Old, No Replies

mainstream stupidity

2:09 am on Nov 21, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
votes: 0

Is it the blind leading the blind [hotwired.lycos.com]?

"For those of you who don't know, search engines use programs
known as spiders or crawlers to index content on Web sites.
Crawlers usually avoid pages with URLs that contain /cgi-bin or
bin (and other variants), as well as CGI escape characters like &,
?, =, etc. These pages often lead to massive databases with
recursive links that can easily trap crawlers in a maze of data.
Sometimes this poses a threat to the crawler, but more often
than not, a trapped spider will simply bring a server to its knees
in a couple of minutes. Although database creators usually don't
intend to trap search-engine crawlers, Bad People sometimes
create recursive crawlspaces for no other reason than pure
mischief. Generally, it's just a bad idea for a spider to follow
such links.

Unfortunately, there's not much you can do right now to get
the crawlers to index your site correctly. It may be tempting to
try to trick them into believing the pages are static HTML, but I
don't recommend this
, since you still run the risk of endangering
your server if the crawler becomes trapped. "

12:37 pm on Nov 23, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 62

It is also from 97 (net bronze age).

I am not too happy about Google walking through all the dynamic pages the way they are. Consider the post you are reading right now has unlimited potential urls. There was one thread here on the system, they indexed at 10 different urls - that isn't right.

1:51 pm on Nov 23, 2000 (gmt 0)

Preferred Member

10+ Year Member

joined:July 28, 2000
votes: 0

Google has got my WMW user profile indexed. The brightside is that it is +1 on the link pop score :)
11:47 am on Nov 24, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
votes: 0

I'm listed too - AV. My ISP conked out, so I logged on using AV free access, of course starting on their home page. The huge ad banner wouldn't budge, so I couldn't type in a URL.

Fortunately, I remembered the thread about people's strange surfing habits, so I typed in this URL in the search box to click on the link to get here, like I had learned. Five results came up. I clicked on #5, and there was a sign-in page with my name on it.