Welcome to WebmasterWorld Guest from 54.227.83.19

Forum Moderators: open

Message Too Old, No Replies

Will inktomi deep crawl for one fee?

does the $39 fee mean slurp will (likely) index all pages?

     

matthew

2:53 am on Dec 9, 2002 (gmt 0)



I was wondering if I pay for the inktomis fast spidering for my index page, does this mean it will deep crawl most likely? Also, off topic a bit, but does having no robot.txt file hurt my "likeability" or chance of being deep crawled/crawled by a spider? thanks.

JayC

7:05 am on Dec 9, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



1) No. The "free" crawl won't happen as a result of your paying for spidering of the index page. Wouldn't make sense, would it? They want you to pay for all of your pages!

2) No. If you don't have a robots.txt you're allowing everybody. No search engine will penalize you for it.

Big_Balou

7:20 am on Dec 9, 2002 (gmt 0)

10+ Year Member



Welcome to Webmaster World.

By paying for the index page INK will only spider that page on the 48hr refresh setup. The rest will be indexed in the normal round of updates. I'm not sure if there is any evidence of having a page in the express search submit will speed up the indexing of the rest of your site. If there is someone will come along to correct me.

At one time there was a "problem" with sites that paid the fee for the quick refresh not having the rest of their pages spidered and indexed in the normal INK update but that seems to have gone away now.

Not having a robots.txt file will not hurt you but is a good idea if there are any files or areas that you do not want indexed. Without one basically a spider sees anything there, at the site, is fair game to be crawled and indexed.

If you want to find out more about the robots.txt file click on the site search link at the top of the window and do a search for it.

You'll find that a lot of the questions you might have have already been covered in some detail and you can find those discussions using the site search....plus a lot of other goodies ;)

tigger

8:20 am on Dec 9, 2002 (gmt 0)

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I do remember sometime ago that index pages were not ranking well under the paid inclusion program, is anyone still finding this?

Big_Balou

11:24 pm on Dec 9, 2002 (gmt 0)

10+ Year Member



Actually for me it is just the opposite. The index page of one site that I used the express deal on is ranked high on multiple KW's while another page, also submited in the express deal, on the same site is buried.

ulstrup

12:05 am on Dec 10, 2002 (gmt 0)

10+ Year Member



For me it's subpages ranking good, index page low, but the competition on the index page keyword is much higher, so I don't think it's because it's the index page
 

Featured Threads

Hot Threads This Week

Hot Threads This Month