goodroi
msg:4098954 | 6:16 pm on Mar 16, 2010 (gmt 0) |
#1 It depends on the situation.
#2 For some situations it can definitely help to block the spiders from reaching certain links & pages. I have done it on a few projects and have been rewarded with better rankings & traffic.
|
dstiles
msg:4099086 | 9:32 pm on Mar 16, 2010 (gmt 0) |
All my web sites (except where a customer requests otherwise) block or remove links from certain pages (eg contact forms) if a bot of any kind is detected, in addition to the page being blocked in robots.txt. If the bot hits the unlinked page then it gets a 405 returned.
|
jonny0000
msg:4099393 | 9:09 am on Mar 17, 2010 (gmt 0) |
Thanks dstiles. I am thinking about preventing access through certain channels rather than preventing access to certain pages altogether where of course robots protocol could be used.
goodroi, what methods have you tested when preventing access through certain links and which have you seen the best results with?
|
jameswsparker
msg:4100173 | 7:47 am on Mar 18, 2010 (gmt 0) |
You can request Google to crawl specific sites/links for it's search engine by adding the URL into Google:
[google.co.uk...]
Entering your site here will get Google to crawl all the links it can find on that site.
|
goodroi
msg:4100276 | 12:15 pm on Mar 18, 2010 (gmt 0) |
every situation is different. i have personally used robots.txt, metarobots and a few others. the method i use depends on the scale of the project and the preexisting structure.
|
|
