Welcome to WebmasterWorld Guest from 54.158.36.59

Forum Moderators: goodroi

Message Too Old, No Replies

Need advice on my robots.txt file

new to robots.txt, can someone check this file

     
12:34 am on Aug 22, 2006 (gmt 0)

5+ Year Member




I'm new to writing robots.txt files, I've read a few tutorials and still not sure about my robots.txt. My question is, do I still need this code [User-agent: * Disallow:] in the robots.txt file to give permission to the major search engines to index my site and is having that in there giving them permission to ignore the rest of the text file below it?

This is what's in my file:

User-agent: *
Disallow:

User-agent: Titan
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-Agent: Googlebot-Image
Disallow: /images/

User-Agent: *
Disallow: /cgi-bin/
Disallow: /encrypt/
Disallow: /gotrythis/
Disallow: /rank/

User-Agent: Scooter
Disallow: /

Thanks for any help,
Jim

3:54 am on Aug 22, 2006 (gmt 0)

10+ Year Member



You have two "User-Agent: *" blocks. You should get rid of the first one, otherwise it will allow spiders to crawl the URLs you've disallowed in the second block.
4:53 am on Aug 22, 2006 (gmt 0)

5+ Year Member



Google has a robot.txt checker at [google.com...] you might want to try that.
And I think your first line is best advised removed.
User-agent: *
Disallow:

The above mentioned means to disallow all bots from indexing your site.
Unless you dont care about getting ranked in the SERPS you can leave that there :)

5:07 pm on Aug 22, 2006 (gmt 0)

5+ Year Member



Thanks abates for the info, it's very helpful.

And to bicycling who wrote:
====================================
Google has a robot.txt checker at [google.com...] you might want to try that.
And I think your first line is best advised removed.
User-agent: *
Disallow:

The above mentioned means to disallow all bots from indexing your site.
Unless you dont care about getting ranked in the SERPS you can leave that there :)
=====================================

I would like to thank you for your reply bicycling and I beleive the block above doesn't request the robots not to index but to index the site, this is the block that disallows:

User-agent: *
Disallow: /

There has to be a / in the block to disallow spiders from indexing, that is what everybody else is saying on the net. Also thanks for the location of that tool.

Jim

 

Featured Threads

Hot Threads This Week

Hot Threads This Month