Welcome to WebmasterWorld Guest from 54.158.195.220

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Google and robots.txt

     
5:43 am on Apr 3, 2007 (gmt 0)

New User

10+ Year Member

joined:Mar 4, 2006
posts:22
votes: 0


How do you get Google to not crawl pages banned in robots.txt?
This is what I have now, but according to webmaster tools it is not seeing these pages as banned. These are search pages, one for each section and they do have noindex, nofollow on them but I would rather not have them crawled. Is there something wrong with my robots.txt syntax? Here is what I have at the end of my robots.txt:

User-agent: *
Disallow: /folder1/Search.asp
Disallow: /folder2/Search.asp
Disallow: /folder3/Search.asp

Thanks

Vortech

3:16 pm on Apr 5, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2005
posts:2259
votes: 0


You can add a noindex meta tag as well. Some people report that the double message gets through.