Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Google crawl

It has started On the 4th?



10:03 pm on Oct 4, 2002 (gmt 0)

10+ Year Member

Started on the 4th this month? Or am I just getting a late crawl?

Just looked at my weblogs and they are crawling my raw directory.

Threw a dummy index.htm file in all of my directories to try to prevent this from happening again.

Hope it works....or maybe I should just keep my server clean.....



10:10 pm on Oct 4, 2002 (gmt 0)

10+ Year Member

There isn't any special crawl going on. The Googlebot can and will spider your files all month long, so don't be surprised when you see her tracks in your logs.

[edited by: dukeblue219 at 11:43 pm (utc) on Oct. 4, 2002]


10:12 pm on Oct 4, 2002 (gmt 0)

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

It's not unusual for sites to still be getting crawled.

Always keep sites clean, I understand a directory index that shows the file structure only is a hole that can be exploited. You can do pages with links to other parts of the site or the index page and use <meta name="robots" content="noindex,nofollow">

The October Everflux thread [webmasterworld.com] already started. Let us know if you see any immediate changes as a result of this, we like to keep track.


Featured Threads

Hot Threads This Week

Hot Threads This Month