Welcome to WebmasterWorld Guest from 54.144.68.27

Forum Moderators: goodroi

Message Too Old, No Replies

Have I been blocking Altavista spider from spidering my site?

     

irock

5:25 pm on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have the following in my robots.txt

User-agent: scooter
Disallow: /

Have I been blocking Altavista spider from spidering my site?

Thanks!

Robino

5:31 pm on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Looks like it.

ncw164x

5:33 pm on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes you have stopped scooter from accessing your site

A few examples of what you would put in your robots.txt file

To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /misc/sitestats/

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

hope this helps

ncw164x

sem4u

5:57 pm on Jan 2, 2004 (gmt 0)

WebmasterWorld Senior Member sem4u is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Yes it does.

ncw164x has given you some good advice there about robots.txt

irock

2:18 am on Jan 5, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Now that I have 'invited' Scooter. When will Altavista put me back on the index?

Thanks!

ncw164x

6:10 pm on Jan 9, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hard to say, keep checking your site stats for any sign of scooter

ncw164x

 

Featured Threads

Hot Threads This Week

Hot Threads This Month