Forum Moderators: goodroi

Message Too Old, No Replies

Can excluding with one word also exclude phrases?

         

Marcia

9:42 pm on Aug 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If I've got an exclusion in robots.txt for

/widgets

will Google also not crawl files in a directory called

/widgets-two

and assume that a directory with anything following just /widget is excluded by default as well, even though the directory with the whole phrase isn't intended to be excluded?

/widgets was crawled when put up, and does show PR. Then I moved the contents to /widgets-two and Google will not touch what's in /widgets-two - all those pages are URL-only and I assume if it was a problem with the pages themselves they'd more likely just crawl them and slap them into the Supplemental index.

I think I may have done myself in by doing just that very thing, with a directory on a site.

Dijkgraaf

10:26 pm on Aug 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, google will not crawl anything begining with /widgits including /widgits-two

Marcia

10:47 pm on Aug 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Duhhh....

I've been aggravating myself and thinking Google took me for a scraper and it just dawned on me this morning to check robots.txt So I did it myself, glad I asked.

Thanks!