homepage Welcome to WebmasterWorld Guest from 54.161.192.61
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Regarding robots.txt file
What does Disallow: /*? mean?
mt_biker

10+ Year Member



 
Msg#: 4252422 posted 4:25 pm on Jan 12, 2011 (gmt 0)

I can't seem to find the answer to this one anywhere...

In this robots.txt file, does anyone know what the following Disallow line means:

User-agent: *
Disallow: /*?

Other than a Sitemap: line, this is the entire robots.txt.

 

mt_biker

10+ Year Member



 
Msg#: 4252422 posted 9:42 pm on Jan 12, 2011 (gmt 0)

Just to clarify my initial question...

I get the Disallow: /* part (don't index anything), but what is the meaning of the question mark?

topr8

WebmasterWorld Senior Member topr8 us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4252422 posted 12:52 am on Jan 13, 2011 (gmt 0)


Disallow: /

means disallow the root and anything below it - effectively disallow all.

what they are saying is:
disallow anything that has a query string

eg
/page.php?a=b

mt_biker

10+ Year Member



 
Msg#: 4252422 posted 3:54 pm on Jan 13, 2011 (gmt 0)

That makes sense. Thank you!

giggle

10+ Year Member



 
Msg#: 4252422 posted 11:33 am on Jan 14, 2011 (gmt 0)

We have this:

User-agent: Googlebot
Disallow: /*?

What does it mean please? I tried reading through various pages but couldn't find the exact answer.

Thanks

aakk9999

WebmasterWorld Administrator 5+ Year Member



 
Msg#: 4252422 posted 1:08 pm on Jan 14, 2011 (gmt 0)

We have this:

User-agent: Googlebot
Disallow: /*?

What does it mean please? I tried reading through various pages but couldn't find the exact answer.


As said in the post above, this means:

what they are saying is:
disallow anything that has a query string

eg
/page.php?a=b

mt_biker

10+ Year Member



 
Msg#: 4252422 posted 4:53 pm on Jan 14, 2011 (gmt 0)

User-agent: *

...means ALL bots whereas...

User-agent: Googlebot

...specifies the Googlebot only

coopster

WebmasterWorld Administrator coopster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4252422 posted 5:42 pm on Jan 14, 2011 (gmt 0)

You may also want to review the forum "Charter" link at the top of this page. Each forum has a Charter with information pertinent to the topics and discussion on the forum. This particular forum has links and tutorials that answer many of your questions regarding the robots.txt directives.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved