homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

bing indexing without crawling
non-spidered pages in serps

WebmasterWorld Senior Member 10+ Year Member

Msg#: 3925920 posted 8:19 pm on Jun 3, 2009 (gmt 0)

For a particular site that only allows GoogleBot to spider, I am seeing Bing has thousands of pages in its index, but the pages have no "blurb" content. (Yahoo has only a handful of pages indexed for this site.)

The pages even appear high in the SERPS for appropriate specific search phrases.

At first glance I thought bing was ignoring robots.txt, but this does not seem to be the case. They must be indexing these pages based on external links.

Has anyone else seen this?



WebmasterWorld Senior Member ogletree us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 3925920 posted 3:01 pm on Jun 8, 2009 (gmt 0)

Most ranking factors are off page anyhow. Try using the noindex tags on your site. Just set up a script that shows the noindex meta tag when a user is not googlebot.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved