Welcome to WebmasterWorld Guest from 54.145.65.62

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt help

     
1:05 pm on Jan 15, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 7, 2003
posts:105
votes: 0


Google seems to be ignoring my robots.txt file am I doing something wrong?

--robots.txt---
User-agent: googlebot
Disallow: CCS/PFCatalog.php

User-agent: googlebot
Disallow: PCR/Catalog/PFCatalog.php

-- from log----
XX.68.82.144 - - [15/Jan/2004:06:17:28] "GET /http://www.******.com/CCS/PFCatalog.php?Cat=90 HTTP/1.1 " 200 200 "None" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

XX.68.82.181 - - [15/Jan/2004:06:25:57] "GET /http://www.******.com/CCS/PFCatalog.php?Cat=91 HTTP/1.1 " 200 200 "None" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

[edited by: DaveAtIFG at 4:58 pm (utc) on Jan. 15, 2004]
[edit reason] "Nuetered" IPs [/edit]

2:16 pm on Jan 15, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 27, 2003
posts:166
votes: 0


Some things to check:

1. Your robots.txt is unusual in that it has more than one record (group of lines separated by a blank line) for the one user-agent googlebot. It might well be that some robots interpret the file in a way that is not what was intended, e.g. only using the last record, or the first record, for Googlebot. Does your robots.txt include still more records for Googlebot?

2. The robots.txt specification [robotstxt.org] says:

Disallow

The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html.

/CCS/PFCatalog.php?Cat=90 does not begin with CCS/PFCatalog.php . It begins with /CCS/PFCatalog.php .

You might try if the following works:


User-agent: *
insert paths to be disallowed to all spiders here

User-agent: googlebot
Disallow: /CCS/PFCatalog.php
Disallow: /PCR/Catalog/PFCatalog.php
insert other paths to be disallowed to Googlebot here

2:35 pm on Jan 15, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 7, 2003
posts:105
votes: 0


could I just then use to handle both

User-agent: googlebot
Disallow: /PFCatalog.php

8:47 am on Jan 16, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 14, 2003
posts:1550
votes: 0


Hi mikemcs,

no because the first / means that you go to the root of the site and your PFCatalog.php seems to be in directories below the root level.

You have to include the full path to the file/directory you want to be ignored.

Hope this helps

Leo

1:04 pm on Jan 16, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 7, 2003
posts:105
votes: 0


Thanks I will give it a try