homepage Welcome to WebmasterWorld Guest from 54.161.166.171
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Regarding robots.txt file
What does Disallow: /*? mean?
mt_biker




msg:4252424
 4:25 pm on Jan 12, 2011 (gmt 0)

I can't seem to find the answer to this one anywhere...

In this robots.txt file, does anyone know what the following Disallow line means:

User-agent: *
Disallow: /*?

Other than a Sitemap: line, this is the entire robots.txt.

 

mt_biker




msg:4252594
 9:42 pm on Jan 12, 2011 (gmt 0)

Just to clarify my initial question...

I get the Disallow: /* part (don't index anything), but what is the meaning of the question mark?

topr8




msg:4252651
 12:52 am on Jan 13, 2011 (gmt 0)


Disallow: /

means disallow the root and anything below it - effectively disallow all.

what they are saying is:
disallow anything that has a query string

eg
/page.php?a=b

mt_biker




msg:4252877
 3:54 pm on Jan 13, 2011 (gmt 0)

That makes sense. Thank you!

giggle




msg:4253229
 11:33 am on Jan 14, 2011 (gmt 0)

We have this:

User-agent: Googlebot
Disallow: /*?

What does it mean please? I tried reading through various pages but couldn't find the exact answer.

Thanks

aakk9999




msg:4253247
 1:08 pm on Jan 14, 2011 (gmt 0)

We have this:

User-agent: Googlebot
Disallow: /*?

What does it mean please? I tried reading through various pages but couldn't find the exact answer.


As said in the post above, this means:

what they are saying is:
disallow anything that has a query string

eg
/page.php?a=b

mt_biker




msg:4253363
 4:53 pm on Jan 14, 2011 (gmt 0)

User-agent: *

...means ALL bots whereas...

User-agent: Googlebot

...specifies the Googlebot only

coopster




msg:4253400
 5:42 pm on Jan 14, 2011 (gmt 0)

You may also want to review the forum "Charter" link at the top of this page. Each forum has a Charter with information pertinent to the topics and discussion on the forum. This particular forum has links and tutorials that answer many of your questions regarding the robots.txt directives.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved