Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: goodroi

Message Too Old, No Replies

Need advice on my robots.txt file

new to robots.txt, can someone check this file

     
12:34 am on Aug 22, 2006 (gmt 0)

New User

5+ Year Member

joined:Aug 14, 2006
posts:5
votes: 0



I'm new to writing robots.txt files, I've read a few tutorials and still not sure about my robots.txt. My question is, do I still need this code [User-agent: * Disallow:] in the robots.txt file to give permission to the major search engines to index my site and is having that in there giving them permission to ignore the rest of the text file below it?

This is what's in my file:

User-agent: *
Disallow:

User-agent: Titan
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-Agent: Googlebot-Image
Disallow: /images/

User-Agent: *
Disallow: /cgi-bin/
Disallow: /encrypt/
Disallow: /gotrythis/
Disallow: /rank/

User-Agent: Scooter
Disallow: /

Thanks for any help,
Jim

3:54 am on Aug 22, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Aug 22, 2003
posts:333
votes: 0


You have two "User-Agent: *" blocks. You should get rid of the first one, otherwise it will allow spiders to crawl the URLs you've disallowed in the second block.
4:53 am on Aug 22, 2006 (gmt 0)

New User

5+ Year Member

joined:Aug 16, 2006
posts:34
votes: 0


Google has a robot.txt checker at [google.com...] you might want to try that.
And I think your first line is best advised removed.
User-agent: *
Disallow:

The above mentioned means to disallow all bots from indexing your site.
Unless you dont care about getting ranked in the SERPS you can leave that there :)

5:07 pm on Aug 22, 2006 (gmt 0)

New User

5+ Year Member

joined:Aug 14, 2006
posts:5
votes: 0


Thanks abates for the info, it's very helpful.

And to bicycling who wrote:
====================================
Google has a robot.txt checker at [google.com...] you might want to try that.
And I think your first line is best advised removed.
User-agent: *
Disallow:

The above mentioned means to disallow all bots from indexing your site.
Unless you dont care about getting ranked in the SERPS you can leave that there :)
=====================================

I would like to thank you for your reply bicycling and I beleive the block above doesn't request the robots not to index but to index the site, this is the block that disallows:

User-agent: *
Disallow: /

There has to be a / in the block to disallow spiders from indexing, that is what everybody else is saying on the net. Also thanks for the location of that tool.

Jim

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members