I can't seem to find the answer to this one anywhere...
In this robots.txt file, does anyone know what the following Disallow line means:
User-agent: * Disallow: /*?
Other than a Sitemap: line, this is the entire robots.txt.
mt_biker
9:42 pm on Jan 12, 2011 (gmt 0)
Just to clarify my initial question...
I get the Disallow: /* part (don't index anything), but what is the meaning of the question mark?
topr8
12:52 am on Jan 13, 2011 (gmt 0)
Disallow: /
means disallow the root and anything below it - effectively disallow all.
what they are saying is: disallow anything that has a query string
eg /page.php?a=b
mt_biker
3:54 pm on Jan 13, 2011 (gmt 0)
That makes sense. Thank you!
giggle
11:33 am on Jan 14, 2011 (gmt 0)
We have this:
User-agent: Googlebot Disallow: /*?
What does it mean please? I tried reading through various pages but couldn't find the exact answer.
Thanks
aakk9999
1:08 pm on Jan 14, 2011 (gmt 0)
We have this:
User-agent: Googlebot Disallow: /*?
What does it mean please? I tried reading through various pages but couldn't find the exact answer.
As said in the post above, this means:
what they are saying is: disallow anything that has a query string
eg /page.php?a=b
mt_biker
4:53 pm on Jan 14, 2011 (gmt 0)
User-agent: *
...means ALL bots whereas...
User-agent: Googlebot
...specifies the Googlebot only
coopster
5:42 pm on Jan 14, 2011 (gmt 0)
You may also want to review the forum "Charter" link at the top of this page. Each forum has a Charter with information pertinent to the topics and discussion on the forum. This particular forum has links and tutorials that answer many of your questions regarding the robots.txt directives.