Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

bing indexing without crawling

non-spidered pages in serps

8:19 pm on Jun 3, 2009 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 1, 2003
votes: 0

For a particular site that only allows GoogleBot to spider, I am seeing Bing has thousands of pages in its index, but the pages have no "blurb" content. (Yahoo has only a handful of pages indexed for this site.)

The pages even appear high in the SERPS for appropriate specific search phrases.

At first glance I thought bing was ignoring robots.txt, but this does not seem to be the case. They must be indexing these pages based on external links.

Has anyone else seen this?

3:01 pm on June 8, 2009 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 14, 2003
votes: 35

Most ranking factors are off page anyhow. Try using the noindex tags on your site. Just set up a script that shows the noindex meta tag when a user is not googlebot.