Forum Moderators: open
This is my first post here, so hopefully its in the right place. I have some questions about Inktomi and AV that hopefully some smart people here can answer.
My site is quite new and also quite large. My site got listed in Zeal back in August. Inktomi (slurp) has been a regular visitor to my site since. Almost daily visits, in fact. Yet, it never progresses beyond the home page. It first looks for a robots.txt file, something I don't have. It then progresses to the index.htm page and goes no further. None of the other spiders (googlebot, av or fast search) seem to have any problem spiering my site - only Inktomi. My thought was that I need to create a robot.txt file specifically for Inktomi. Any ideas on this? And if I do need to create a robots.txt file, how should I create it? I have no idea how to create a robots.txt file that won't mess up anything else. Step by step instructions would be helpful! All my metatage are set to have the indexes follow all links - I did this specifically for the benefit of Inktomi but without any success.
Also, both AV and Looksmart(wisenut) have been at my site alot, too. Problem is, in Wisenut my site is still nowhere to be found. In AV, my site is only showing 10 pages (out of a total of 700 that I have). Is this very slow introduction into AV and Wisenut normal?
Finally, Fast Search also visits regularly and takes in alot of pages when it does. How often does Fast Search update its index? The listings it has for my site show docuements that are easily 2+ months old, even though Fast has been at my site alot over the past 4 weeks or so.
Thanks.
Jim
You really need to have a robots.txt. here is a tutorial to help you build one:
[searchengineworld.com...]
Here is also a very good faq on fast
[webmasterworld.com...]
For me atleast AV and Wisenut don't even cross my radar. I really don't know how much traffic they get. For some phrases that I do very well in google, I am number one at both AV and Wisenut. I see almost no traffic from those searches.
I've read here and there that the default (what the spider will do in the absence of either robots.txt or robots meta tags on page) for Ink is "index, nofollow", whereas for any other spider it is "index, follow".
"index, nofollow" doesn't seem to make much sense until you think about their PFI focus. This default would be a logical way to avoid indexing much free content.
Either way, it wouldn't hurt to explicitly tell Ink to "index,follow".
Thanks for the link to the robots.txt area. So to get all spiders into my site, all I have to do to create a robots.txt by using Notepad and putting this in?
User-agent: *
Disallow:
Is this all there is to a robots.txt file? Or do I have to convert it to something else before I upload it? And where to I put this file?
Sorry for the stupid questions! Just want to do it right lest Google or other spiders take offense.
Thanks.
Jim
