Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt help

1:05 pm on Jan 15, 2004 (gmt 0)

10+ Year Member

Google seems to be ignoring my robots.txt file am I doing something wrong?

User-agent: googlebot
Disallow: CCS/PFCatalog.php

User-agent: googlebot
Disallow: PCR/Catalog/PFCatalog.php

-- from log----
XX.68.82.144 - - [15/Jan/2004:06:17:28] "GET /http://www.******.com/CCS/PFCatalog.php?Cat=90 HTTP/1.1 " 200 200 "None" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

XX.68.82.181 - - [15/Jan/2004:06:25:57] "GET /http://www.******.com/CCS/PFCatalog.php?Cat=91 HTTP/1.1 " 200 200 "None" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

[edited by: DaveAtIFG at 4:58 pm (utc) on Jan. 15, 2004]
[edit reason] "Nuetered" IPs [/edit]

2:16 pm on Jan 15, 2004 (gmt 0)

10+ Year Member

Some things to check:

1. Your robots.txt is unusual in that it has more than one record (group of lines separated by a blank line) for the one user-agent googlebot. It might well be that some robots interpret the file in a way that is not what was intended, e.g. only using the last record, or the first record, for Googlebot. Does your robots.txt include still more records for Googlebot?

2. The robots.txt specification [robotstxt.org] says:


The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html.

/CCS/PFCatalog.php?Cat=90 does not begin with CCS/PFCatalog.php . It begins with /CCS/PFCatalog.php .

You might try if the following works:

User-agent: *
insert paths to be disallowed to all spiders here

User-agent: googlebot
Disallow: /CCS/PFCatalog.php
Disallow: /PCR/Catalog/PFCatalog.php
insert other paths to be disallowed to Googlebot here

2:35 pm on Jan 15, 2004 (gmt 0)

10+ Year Member

could I just then use to handle both

User-agent: googlebot
Disallow: /PFCatalog.php

8:47 am on Jan 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Hi mikemcs,

no because the first / means that you go to the root of the site and your PFCatalog.php seems to be in directories below the root level.

You have to include the full path to the file/directory you want to be ignored.

Hope this helps


1:04 pm on Jan 16, 2004 (gmt 0)

10+ Year Member

Thanks I will give it a try