robots.txt Help

disallow issue

[malasorte]

User-agent: *
Disallow: dir1/dir2

Does that mean robots will crawll anything that is in dir1, except dir2?

[jimbeetle]

Nope. It means that anything in dir1/dir2 will not be crawled. That is if the bot understands the directive with the root (/) not being indicated.

User-agent: *
Disallow: /dir1/dir2/

Depending on the structure of dir1 you can implicity allow dir2 by not disallowing it:

User-agent: *
Disallow: /dir1/dir3/
Disallow: /dir1/dir4/
Disallow: /dir1/dir5/

But, this also implicity allows any pages in dir1 itself, such as /dir1/example.htm.

More info at robotstxt.org [robotstxt.org].

[malasorte]

jimbeetle thanks for the expert advice!