homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Disallow Yahoo to block site explorer

 10:47 pm on Aug 14, 2008 (gmt 0)

Has anyone blocked yahoo slurp to hide backlinks from Yahoo! Site Explorer.

The reason I ask, I am wondering if Yahoo!
1. removes old backlinks from Site Explorer searches.
[ I know they are slow to remove old link data, but do they eventually remove it? ]
2. honors the disallow when backlinks are considered in general.

So far I have not seen a much of impact of disallowing link pages from Yahoo! in terms of rankings to target sites. However it has only been a month since I have disallowed Y!.

I would think I should know in about 5 months, but if anyone is willing to give up their personal past experiences, I will follow up with mine.



 2:08 pm on Aug 15, 2008 (gmt 0)

If you are looking for a site that has blocked Yahoo (and MSN) then check out schools.nyc.gov/robots.txt. They are not doing for link issues like you but for bandwidth issues (which imho is silly).

You do bring up a good point about disallowing links from Yahoo. Historically they have treated nofollow tags differently than Yahoo. I would expect Yahoo to respect a robots.txt disallow. It just might take some time for them to make the change (especially if you are disallowing a very deep page). Once they make the change will they remove the link data from their database or leave it their since slurp is blocked from confirming the links are removed? Looking forward to figuring this one out.

Lord Majestic

 2:16 pm on Aug 15, 2008 (gmt 0)

This (if it works at all) will only hide internal backlinks from your site in Site Explorer - if you are afraid of the competition then it is the external links to your site that they would be most interested in anyway and those are not under your control (unless they come from your own additional sites).

In all probability (and I'd say certainty) backlinks index is a separate one that gets additions from recent crawls - this means that if you prevent your site from being crawled then the best you can achieve is that no new backlinks (from your sites) will be added.

Yahoo shows nofollow marked links in Site Explorer just fine (Google appears to do the same).


 10:08 pm on Aug 24, 2008 (gmt 0)

To this date, any new links placed on sites that have been disallowed, do not show up when using site explorer.

Old link data still exists.
I will repost if old data is flushed.


 2:12 am on Nov 11, 2008 (gmt 0)

four months to flush

Lord Majestic

 2:31 am on Nov 11, 2008 (gmt 0)

Do you still see external backlinks shown for your site found on websites where you did not disallow YSE for whatever reason (like when the website owned by other people)?


 6:17 pm on Nov 11, 2008 (gmt 0)


Lord Majestic

 6:56 pm on Nov 11, 2008 (gmt 0)

Ok, so in effect all you achieved is removed from YSE backlinks founds on sites that you personally control and it took almost 4 months to do so, where as external backlinks from other sites not under your control are still being shown in YSE.


 2:23 am on Nov 14, 2008 (gmt 0)

Lord Majestic
like your bot :)
can you comp me an extended trial?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved