Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Unknowingly restricting some directories

6:20 am on Jan 13, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 27, 2002
votes: 0

Hi, for the longest time I have been restricting some important directories on my site from being crawled. My mistake was simple (and superiorly dumb). I left out the slash (/) after the name of the directory.


User-agent: *
Disallow: /b

I found out that by not putting a slash after the b, I am restricting all directories starting with that letter! ARgh!

Now that I have corrected it, will it still be possible to get Google et al to include the once restricted directories? How long should that take? I am now linking heavily to the pages in question and they are included in my sitemap. Please tell me what other things I can do. Thanks!

4:40 am on Jan 20, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 31, 2005
votes: 0

Your aren't only disallowing any directories starting with b, but everything starting with b.

See also this thread [webmasterworld.com] about why it is sometimes not a good idea to have a trailing slash for disallowing directories.

I'd suggest making your directory names more meaningfull and distinct from any other pages/resources, so your disallow can refer only to those things you want to disallow.

Just make the changes to your robots.txt and naming conventions and they will get indexed eventually.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members