Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo now violating robots.txt


Marcia - 3:46 am on Dec 29, 2007 (gmt 0)


I believe it was something I read that was written by Nathan Buggia on their blog, possibly at their in-house forum, too.

OK, I re-read the post by MSNdude. They are too! indexing pages with a robots noindex, nofollow. Also grabbing links off pages and indexing those LINKS instead of the page itself.

Is all that that confused, rude or accidental?

This is both MSN and Yahoo both, tons of them

[webmasterworld.com...]

[edited by: Marcia at 3:51 am (utc) on Dec. 29, 2007]


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3536187.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com