| Google crawl
It has started On the 4th? |
zuko105
msg:112625 | 10:03 pm on Oct 4, 2002 (gmt 0) |
Started on the 4th this month? Or am I just getting a late crawl? Just looked at my weblogs and they are crawling my raw directory. Threw a dummy index.htm file in all of my directories to try to prevent this from happening again. Hope it works....or maybe I should just keep my server clean..... Zuko
|
dukeblue219
msg:112626 | 10:10 pm on Oct 4, 2002 (gmt 0) |
There isn't any special crawl going on. The Googlebot can and will spider your files all month long, so don't be surprised when you see her tracks in your logs. [edited by: dukeblue219 at 11:43 pm (utc) on Oct. 4, 2002]
|
Marcia
msg:112627 | 10:12 pm on Oct 4, 2002 (gmt 0) |
It's not unusual for sites to still be getting crawled. Always keep sites clean, I understand a directory index that shows the file structure only is a hole that can be exploited. You can do pages with links to other parts of the site or the index page and use <meta name="robots" content="noindex,nofollow"> The October Everflux thread [webmasterworld.com] already started. Let us know if you see any immediate changes as a result of this, we like to keep track.
|
|
|
