homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

mainstream stupidity

 2:09 am on Nov 21, 2000 (gmt 0)

Is it the blind leading the blind [hotwired.lycos.com]?

"For those of you who don't know, search engines use programs
known as spiders or crawlers to index content on Web sites.
Crawlers usually avoid pages with URLs that contain /cgi-bin or
bin (and other variants), as well as CGI escape characters like &,
?, =, etc. These pages often lead to massive databases with
recursive links that can easily trap crawlers in a maze of data.
Sometimes this poses a threat to the crawler, but more often
than not, a trapped spider will simply bring a server to its knees
in a couple of minutes. Although database creators usually don't
intend to trap search-engine crawlers, Bad People sometimes
create recursive crawlspaces for no other reason than pure
mischief. Generally, it's just a bad idea for a spider to follow
such links.

Unfortunately, there's not much you can do right now to get
the crawlers to index your site correctly. It may be tempting to
try to trick them into believing the pages are static HTML, but I
don't recommend this
, since you still run the risk of endangering
your server if the crawler becomes trapped. "



 12:37 pm on Nov 23, 2000 (gmt 0)

It is also from 97 (net bronze age).

I am not too happy about Google walking through all the dynamic pages the way they are. Consider the post you are reading right now has unlimited potential urls. There was one thread here on the system, they indexed at 10 different urls - that isn't right.


 1:51 pm on Nov 23, 2000 (gmt 0)

Google has got my WMW user profile indexed. The brightside is that it is +1 on the link pop score :)


 11:47 am on Nov 24, 2000 (gmt 0)

I'm listed too - AV. My ISP conked out, so I logged on using AV free access, of course starting on their home page. The huge ad banner wouldn't budge, so I couldn't type in a URL.

Fortunately, I remembered the thread about people's strange surfing habits, so I typed in this URL in the search box to click on the link to get here, like I had learned. Five results came up. I clicked on #5, and there was a sign-in page with my name on it.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved