Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: open
Instead of it going to:
It's looking for
The /redwidgets/redwidgets/ directory has never existed.
Has anyone else experienced this problem?
It's not just looking for one page, it's looking for hundreds of pages in a non existent directory. It's like it' stuttering on the directory name.
I have a /widgets/page.html and about 300 more pages in /widgets/. It's trying to find each page that exists in /widgets/ but looking in /widgets/widgets/ instead.
I have MANY 404's in my log (my custom 404 includes a sitemap).
And did you just move some sites up or down a level (that is, from "/" to "/widgets" or from "/widgets/shopping" to "/widgets") or perhaps just around? Or perhaps the next theory is by itself enough to account for it.
And/or you're using FrontPage or .NET or some M$ tool -- my experience is M$ tools are all VERY bad about portability of categories within projects, and of course while nobody else on earth could aspire to duplicate ALL of M$'s program bugs, many other programmers are capable of recreating specific ones; so it could even be a tool from some other source -- and perhaps the root of your website on your machine is something like c:\widgets\ .
I just don't get this redirect thing. It had been grabbing pages for days, then sometime yesterday morning it took my robots.txt (for the umteenth time) and now it's acting bizarre. No other spiders seem affected, Slurp has been busy all day, without problems.
I don't know, I just don't know.