Welcome to WebmasterWorld Guest from 54.197.171.28

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt

Can you exclude a parameter in robots.txt?

   
1:49 pm on May 15, 2007 (gmt 0)

5+ Year Member



Can I exclude a parameter in a robots.txt? For example,

User-agent: *
Disallow: *path=*

2:23 pm on May 15, 2007 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Both Google [google.com] and Yahoo [help.yahoo.com] support wildcards (and to a certain extent, for file formats, also Live). I would give very serious thought before implementing this because of the lack of support among other bots.
9:22 pm on May 18, 2007 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



You need to put those instructions in a section of robots.txt especially for Googlebot.

Be aware that if the Googlebot section is present, Google does NOT read any other sections of the robots.txt file; especially the User-agent: * section of the file.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month