Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
My site dropped out of Inktomi in August last year and hasn't had a single page spidered by slurp since. I lost around 20% of my traffic as a result of this drop!
These last couple of days slurp has been requesting my robots file around 8 - 10 times a day!
Has anyone else had this experience? Could this mean that my site is on its way back in to Ink?
This is my robotfile:
As well I did save it in linux mode and uploaded it in ascii or something like that, to make sure itīs correctly, but they keep on spidering only the robots.txt. I have another post on it named why does Inktomi spiders me, where you can see I wrote them and they gave me their link to their websearch-guidelines.............
They said, but donīt know why, that when slurp spider the robots.txt file is always like this:
"Mozilla/5.0 (Slurp/si; firstname.lastname@example.org; [inktomi.com...]
and when spider rest of site ítīs like this:
"Mozilla/5.0 (Slurp/cat; email@example.com; [inktomi.com...]
The diference are in Slurp/si; and Slurp/cat,
just checked, all my spidered robots file are with slurp/si
That means there are two diferent spiders?
and that slurp/cat maybe not started to work properly yet?
My robots.txt is very basic and valid and it doesn't cause problems for other bots so I don't think its keeping slurp out.
I checked another sites logs and found that slurp is indexing pages with (Slurp/cat) and robots files with (Slurp/si).
Looks like I need to wait for Slurp/si's big brother to visit. Hopefully it's only a matter of time.