Forum Moderators: goodroi
This is what's in my file:
User-agent: *
Disallow:
User-agent: Titan
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-Agent: Googlebot-Image
Disallow: /images/
User-Agent: *
Disallow: /cgi-bin/
Disallow: /encrypt/
Disallow: /gotrythis/
Disallow: /rank/
User-Agent: Scooter
Disallow: /
Thanks for any help,
Jim
The above mentioned means to disallow all bots from indexing your site.
Unless you dont care about getting ranked in the SERPS you can leave that there :)
Thanks abates for the info, it's very helpful.
And to bicycling who wrote:
====================================
Google has a robot.txt checker at [google.com...] you might want to try that.
And I think your first line is best advised removed.
User-agent: *
Disallow:
The above mentioned means to disallow all bots from indexing your site.
Unless you dont care about getting ranked in the SERPS you can leave that there :)
=====================================
I would like to thank you for your reply bicycling and I beleive the block above doesn't request the robots not to index but to index the site, this is the block that disallows:
User-agent: *
Disallow: /
There has to be a / in the block to disallow spiders from indexing, that is what everybody else is saying on the net. Also thanks for the location of that tool.
Jim