Welcome to WebmasterWorld Guest from 54.197.66.254

Forum Moderators: goodroi

Message Too Old, No Replies

Regarding robots.txt file

What does Disallow: /*? mean?

     
4:25 pm on Jan 12, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2004
posts: 75
votes: 0


I can't seem to find the answer to this one anywhere...

In this robots.txt file, does anyone know what the following Disallow line means:

User-agent: *
Disallow: /*?

Other than a Sitemap: line, this is the entire robots.txt.
9:42 pm on Jan 12, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2004
posts: 75
votes: 0


Just to clarify my initial question...

I get the Disallow: /* part (don't index anything), but what is the meaning of the question mark?
12:52 am on Jan 13, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member topr8 is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 19, 2002
posts:3230
votes: 17



Disallow: /

means disallow the root and anything below it - effectively disallow all.

what they are saying is:
disallow anything that has a query string

eg
/page.php?a=b
3:54 pm on Jan 13, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2004
posts: 75
votes: 0


That makes sense. Thank you!
11:33 am on Jan 14, 2011 (gmt 0)

Preferred Member

10+ Year Member

joined:May 23, 2002
posts: 446
votes: 0


We have this:

User-agent: Googlebot
Disallow: /*?

What does it mean please? I tried reading through various pages but couldn't find the exact answer.

Thanks
1:08 pm on Jan 14, 2011 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Apr 30, 2008
posts:2628
votes: 191


We have this:

User-agent: Googlebot
Disallow: /*?

What does it mean please? I tried reading through various pages but couldn't find the exact answer.


As said in the post above, this means:

what they are saying is:
disallow anything that has a query string

eg
/page.php?a=b
4:53 pm on Jan 14, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2004
posts: 75
votes: 0


User-agent: *

...means ALL bots whereas...

User-agent: Googlebot

...specifies the Googlebot only
5:42 pm on Jan 14, 2011 (gmt 0)

Administrator

WebmasterWorld Administrator coopster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 31, 2003
posts:12541
votes: 1


You may also want to review the forum "Charter" link at the top of this page. Each forum has a Charter with information pertinent to the topics and discussion on the forum. This particular forum has links and tutorials that answer many of your questions regarding the robots.txt directives.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members