Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Google crawl

It has started On the 4th?

10:03 pm on Oct 4, 2002 (gmt 0)

Full Member

10+ Year Member

joined:June 24, 2002
votes: 0

Started on the 4th this month? Or am I just getting a late crawl?

Just looked at my weblogs and they are crawling my raw directory.

Threw a dummy index.htm file in all of my directories to try to prevent this from happening again.

Hope it works....or maybe I should just keep my server clean.....


10:10 pm on Oct 4, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:June 26, 2002
votes: 0

There isn't any special crawl going on. The Googlebot can and will spider your files all month long, so don't be surprised when you see her tracks in your logs.

[edited by: dukeblue219 at 11:43 pm (utc) on Oct. 4, 2002]

10:12 pm on Oct 4, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
votes: 0

It's not unusual for sites to still be getting crawled.

Always keep sites clean, I understand a directory index that shows the file structure only is a hole that can be exploited. You can do pages with links to other parts of the site or the index page and use <meta name="robots" content="noindex,nofollow">

The October Everflux thread [webmasterworld.com] already started. Let us know if you see any immediate changes as a result of this, we like to keep track.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members