Welcome to WebmasterWorld Guest from 54.160.163.163

Forum Moderators: goodroi

Message Too Old, No Replies

What was wrong with this robots.txt if anything

     

montenegro

10:58 am on Apr 15, 2005 (gmt 0)

10+ Year Member



I am just about to start screaming. I have just discovered by accident that my robots.txt file had a huge problems. I checked it before uploading and Search Engine World Robots.txt Validator did not report problems and it still doesn't. This evening I used META TAG ANALYZER available on many sites on the web I got "Error: 403 Forbidden by robots.txt" message for all pages including the mydomain.com/. META TAG ANALYZER returned correct result only for mydomain.com (without the forward slash character).
Here is my robots.txt that I used fir the last 3 years:
User-agent: *
Disallow: /private
Disallow: /cgi-bin
Disallow: /scgi-bin
Disallow: /cpanel-upgrade
User-Agent: sitecheck.internetseer.com
Disallow: /
User-Agent: NPBot-1/2.0
Disallow: /
User-agent: linksmanager
Disallow: /
User-agent: Cyveillance
Disallow: /

After I changed the code to the following every page became visable to the tool (no "Error: 403 Forbidden by robots.txt" returned):
User-agent: *
Disallow: /private
Disallow: /cgi-bin
Disallow: /scgi-bin
Disallow: /cpanel-upgrade
User-Agent: sitecheck.internetseer.com
Disallow: /
User-Agent: NPBot-1/2.0
Disallow: /
User-agent: linksmanager
Disallow: /
User-agent: Cyveillance
Disallow: /

My questions are:
1. Is it possible that because of this my rankings have been affected for the last three years?
2. Is this a bug in the actual tool? All pages on my site are indexed.
ANY COMMENTS?

Reid

6:32 am on Apr 16, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I'd try a few different META analyzer tools maybe this one is using one of the robots you are blocking.

montenegro

8:36 am on Apr 16, 2005 (gmt 0)

10+ Year Member



An important correction to my post. The oroginal post above did not reflect the changes in my robots.txt file.

robots.txt file that I used for 3 years:

User-agent: *
Disallow: /private
Disallow: /cgi-bin
Disallow: /scgi-bin
Disallow: /cpanel-upgrade
User-Agent: sitecheck.internetseer.com
Disallow: /
User-Agent: NPBot-1/2.0
Disallow: /
User-agent: linksmanager
Disallow: /
User-agent: Cyveillance
Disallow: /

After I changed the code to the following every page became visable to the tool (no "Error: 403 Forbidden by robots.txt" returned):

User-agent: *
Disallow: /private
Disallow: /cgi-bin
Disallow: /scgi-bin
Disallow: /cpanel-upgrade

User-Agent: sitecheck.internetseer.com
Disallow: /
User-Agent: NPBot-1/2.0
Disallow: /
User-agent: linksmanager
Disallow: /
User-agent: Cyveillance
Disallow: /

The only change is a blanc line inserted after "Disallow: /cpanel-upgrade"

My questions are:
1. Is it possible that because of this my rankings have been affected for the last three years?
2. Is this a bug in the actual tool? All pages on my site are indexed.
ANY COMMENTS?

Span

9:25 am on Apr 16, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The blank line makes no difference, as far as I know.
You should, however, if you want to exclude directories from being spidered, use a trailing slash:

Disallow: /private/

- if there's no slash at the end bots assume it is a file and they do spider the directory.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month