homepage Welcome to WebmasterWorld Guest from 54.147.248.118
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Inktomi Spider Behavior
what does this mean
graywolf

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 275 posted 3:02 pm on Feb 9, 2004 (gmt 0)

The inktomi spider has come twice in the past 30 days and the only file it looked for was robots.txt. I don't identify it specifically in the robots file could this be why it's not looking for any other pages?

 

Finder

10+ Year Member



 
Msg#: 275 posted 3:59 pm on Feb 16, 2004 (gmt 0)

Not likely. Some bots update their robots cache regularly, regardless of their spidering schedule.

andy_boyd

10+ Year Member



 
Msg#: 275 posted 5:43 pm on Feb 16, 2004 (gmt 0)

For a while there Slurp only came in, looked at the index and robots.txt and left. But for the last week or so Slurp has been all over one of my sites, can't get enough. Strange behaviour.

helenp

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 275 posted 10:45 pm on Feb 16, 2004 (gmt 0)

I have the same problem, since 13/1 until today,
inktomi slurp/si only spiders my robots.txt, and some days several times, one day 14 times.
write twice to Inktomi, but nothing, no good anwers.

helenp

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 275 posted 12:31 am on Feb 17, 2004 (gmt 0)

Just noticied that gigabot also only spideres my robot.txt,
My robots.txt are validated and checked and doubblechecked, the rest of spiders donīt have any problems.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved