| Welcome to WebmasterWorld Guest from 188.8.131.52 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
|Have I been blocking Altavista spider from spidering my site?|
| 5:25 pm on Jan 2, 2004 (gmt 0)|
I have the following in my robots.txt
Have I been blocking Altavista spider from spidering my site?
| 5:31 pm on Jan 2, 2004 (gmt 0)|
Looks like it.
| 5:33 pm on Jan 2, 2004 (gmt 0)|
Yes you have stopped scooter from accessing your site
A few examples of what you would put in your robots.txt file
To exclude all robots from parts of the server
This indicates that nothing is disallowed and the spider can follow all links
To allow a single robot complete access and exclude all others
This would prevent your entire web site from being indexed
hope this helps
| 5:57 pm on Jan 2, 2004 (gmt 0)|
Yes it does.
ncw164x has given you some good advice there about robots.txt
| 2:18 am on Jan 5, 2004 (gmt 0)|
Now that I have 'invited' Scooter. When will Altavista put me back on the index?
| 6:10 pm on Jan 9, 2004 (gmt 0)|
Hard to say, keep checking your site stats for any sign of scooter
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved