Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt not working

Disallows are still being indexed

3:01 pm on Mar 5, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:June 23, 2003
votes: 0

About 5 months ago I added a line to my robots.txt file to stop google from indexing a page. However, this page still exists in Google and googlebot comes and grabs the page every day. Any ideas what I'm doing wrong? Below is my robots.txt file:

User-agent: *
Disallow: PageA.asp

3:19 pm on Mar 5, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 22, 2003
votes: 0

Hi jchance,

Should you not have a forward slash in front of the disallowed page?

User-agent: *
Disallow: /PageA.asp

Apart from that am at a loss :(


5:21 am on Mar 12, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
votes: 0

Or maybe you put the robots.txt file in a directory other than your root web directory.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members