Forum Moderators: open
I would greatly appreciate advice on how i can stop Y's spider from visisting incomplete URL's on my site.
i've a website thats in several langauges. whilst reviewing my log file yesterady i came across some disturbing news.
yahoo's been visiting some incomplete url's
[mysite.com...]
My log file shows that when slurp's crawled this url it recieved a server header response 302 object moved.
if i place the url through the server header checker it returns a 200 ok response this i believe is because it defaults/redirects to the index.htm page.
but if i could i'd like this to be stopped completely as Yahoo now has two pages in its results
[mysite.com...]
and
[mysite.com...]
this i see as being a problem in the near future with the duplicate content filter.
I've used the link command to try and find the offending site that is linking to this short URL with no luck.
What can i do to stop this?
can i set up a server wide 403 forbidden on incomplete url's if so how do i do that?
I understand from reading this forum for a while, that Y has probelms following any redirects, where G hates the 302 respose. i've seen no activity in the log file from G on these URL's but if yahoo gets there redirect problem sorted out, they will/might throw me out of their SERP's
Any advise on this would be greatly appreciate, if this site gets wiped i'm going to be broke.
Vimes
After reviewing my logs again this morning I’m seeing Yahoo hitting multiple incomplete url's,
Is there no way I can stop this?
I’m seeing more and more pages indexed now with incomplete URL's.
I need some help here my SERP's are dropping on these search terms.
Anyone have any suggestions how I can stop this.
Vimes
But I’ve checked the response from my 404 page and it doesn't give a 302 response.
This is taken from their site
(not sure if I’m allowed to post links to msoft)
"IIS generates courtesy redirect when folder without trailing slash is requested"
this redirect is automatic and out of my control?
I'm seeing Slurp's hit the page again with the correct /index.htm but why would they index the page with the incomplete URL.
so is this just one more demonstration on how yahoo is inept in following any redirection, and is there no way i can stop this site from being burned.
I did have some really good SERP's on these terms now that i seem to be penalised because of the dupe content, which actually doesn’t exist!
Vimes
