Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Will inktomi deep crawl for one fee?

does the $39 fee mean slurp will (likely) index all pages?

2:53 am on Dec 9, 2002 (gmt 0)

New User

joined:Mar 9, 2012
votes: 0

I was wondering if I pay for the inktomis fast spidering for my index page, does this mean it will deep crawl most likely? Also, off topic a bit, but does having no robot.txt file hurt my "likeability" or chance of being deep crawled/crawled by a spider? thanks.
7:05 am on Dec 9, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 5, 2001
votes: 0

1) No. The "free" crawl won't happen as a result of your paying for spidering of the index page. Wouldn't make sense, would it? They want you to pay for all of your pages!

2) No. If you don't have a robots.txt you're allowing everybody. No search engine will penalize you for it.

7:20 am on Dec 9, 2002 (gmt 0)

Full Member

10+ Year Member

joined:May 23, 2002
votes: 0

Welcome to Webmaster World.

By paying for the index page INK will only spider that page on the 48hr refresh setup. The rest will be indexed in the normal round of updates. I'm not sure if there is any evidence of having a page in the express search submit will speed up the indexing of the rest of your site. If there is someone will come along to correct me.

At one time there was a "problem" with sites that paid the fee for the quick refresh not having the rest of their pages spidered and indexed in the normal INK update but that seems to have gone away now.

Not having a robots.txt file will not hurt you but is a good idea if there are any files or areas that you do not want indexed. Without one basically a spider sees anything there, at the site, is fair game to be crawled and indexed.

If you want to find out more about the robots.txt file click on the site search link at the top of the window and do a search for it.

You'll find that a lot of the questions you might have have already been covered in some detail and you can find those discussions using the site search....plus a lot of other goodies ;)

8:20 am on Dec 9, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 16, 2000
votes: 0

I do remember sometime ago that index pages were not ranking well under the paid inclusion program, is anyone still finding this?
11:24 pm on Dec 9, 2002 (gmt 0)

Full Member

10+ Year Member

joined:May 23, 2002
votes: 0

Actually for me it is just the opposite. The index page of one site that I used the express deal on is ranked high on multiple KW's while another page, also submited in the express deal, on the same site is buried.
12:05 am on Dec 10, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 25, 1999
votes: 0

For me it's subpages ranking good, index page low, but the competition on the index page keyword is much higher, so I don't think it's because it's the index page

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members