MikeNoLastName - 10:10 pm on Feb 10, 2004 (gmt 0)
We have been having quite a problem with FAST (Lycos, ATW) and submitting our site. We submitted two pages at the same time. One went right through and got indexed fine. The other, our home page, claims it was submitted and ACCEPTED, but the url.all search produces absolutely nothing in either ATW or Lycos. This despite the fact that ATW previously DID index the submitted page as well as about 200 others from the same domain. They are now ALL gone from ATW.
After hours of research it appears that at one time we obtained a duplicate page penalty from FAST/ATW/Lycos, on domain "A" because of a copy of A's home page from years ago having accidently been uploaded to the wrong directory, and being found in a subdirectory of domain "B" (they all still index the copy in the B subdirectory - the two pages are now about 60-70% the same). We discovered the same phenomena on Inktomi, and was able to correct the problem within a couple of weeks, by 301 Redirecting the duplicate on B to the correct, official home page on A and suddenly the B duplicate disappeared from the index and the correct page in A got properly indexed at last. End of problem.
However, on FAST/ATW/Lycos this does not seem to work. I suspect it might be because of the following. If you go to the Insite submit page and select the Spider submit option. And then enter the URL of a, or in this case, THE duplicate page on domain B with the 301 redirect, InSite returns with a message that claims there is an ERROR on the page entered and refuses to spider it! Thus it appears there may not be a simple way to remove the duplicate page from FAST in this case. Lycos's search results show the duplicate page on "B" has not been refreshed since Nov 2003.
I've contacted Lycos & ATW repeatedly, but haven't heard a thing back.