homepage Welcome to WebmasterWorld Guest from 54.198.46.115
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Have I been blocking Altavista spider from spidering my site?
irock




msg:1525702
 5:25 pm on Jan 2, 2004 (gmt 0)

I have the following in my robots.txt

User-agent: scooter
Disallow: /

Have I been blocking Altavista spider from spidering my site?

Thanks!

 

Robino




msg:1525703
 5:31 pm on Jan 2, 2004 (gmt 0)

Looks like it.

ncw164x




msg:1525704
 5:33 pm on Jan 2, 2004 (gmt 0)

Yes you have stopped scooter from accessing your site

A few examples of what you would put in your robots.txt file

To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /misc/sitestats/

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

hope this helps

ncw164x

sem4u




msg:1525705
 5:57 pm on Jan 2, 2004 (gmt 0)

Yes it does.

ncw164x has given you some good advice there about robots.txt

irock




msg:1525706
 2:18 am on Jan 5, 2004 (gmt 0)

Now that I have 'invited' Scooter. When will Altavista put me back on the index?

Thanks!

ncw164x




msg:1525707
 6:10 pm on Jan 9, 2004 (gmt 0)

Hard to say, keep checking your site stats for any sign of scooter

ncw164x

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved