Welcome to WebmasterWorld Guest from 23.20.110.176

Forum Moderators: goodroi

Message Too Old, No Replies

Regarding robots.txt file

What does Disallow: /*? mean?

     
4:25 pm on Jan 12, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2004
posts: 75
votes: 0


I can't seem to find the answer to this one anywhere...

In this robots.txt file, does anyone know what the following Disallow line means:

User-agent: *
Disallow: /*?

Other than a Sitemap: line, this is the entire robots.txt.
9:42 pm on Jan 12, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2004
posts: 75
votes: 0


Just to clarify my initial question...

I get the Disallow: /* part (don't index anything), but what is the meaning of the question mark?
12:52 am on Jan 13, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member topr8 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 19, 2002
posts:3196
votes: 12



Disallow: /

means disallow the root and anything below it - effectively disallow all.

what they are saying is:
disallow anything that has a query string

eg
/page.php?a=b
3:54 pm on Jan 13, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2004
posts: 75
votes: 0


That makes sense. Thank you!
11:33 am on Jan 14, 2011 (gmt 0)

Preferred Member

10+ Year Member

joined:May 23, 2002
posts: 444
votes: 0


We have this:

User-agent: Googlebot
Disallow: /*?

What does it mean please? I tried reading through various pages but couldn't find the exact answer.

Thanks
1:08 pm on Jan 14, 2011 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Apr 30, 2008
posts:2599
votes: 179


We have this:

User-agent: Googlebot
Disallow: /*?

What does it mean please? I tried reading through various pages but couldn't find the exact answer.


As said in the post above, this means:

what they are saying is:
disallow anything that has a query string

eg
/page.php?a=b
4:53 pm on Jan 14, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2004
posts: 75
votes: 0


User-agent: *

...means ALL bots whereas...

User-agent: Googlebot

...specifies the Googlebot only
5:42 pm on Jan 14, 2011 (gmt 0)

Administrator

WebmasterWorld Administrator coopster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 31, 2003
posts:12541
votes: 1


You may also want to review the forum "Charter" link at the top of this page. Each forum has a Charter with information pertinent to the topics and discussion on the forum. This particular forum has links and tutorials that answer many of your questions regarding the robots.txt directives.