homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Will inktomi deep crawl for one fee?
does the $39 fee mean slurp will (likely) index all pages?

Msg#: 1852 posted 2:53 am on Dec 9, 2002 (gmt 0)

I was wondering if I pay for the inktomis fast spidering for my index page, does this mean it will deep crawl most likely? Also, off topic a bit, but does having no robot.txt file hurt my "likeability" or chance of being deep crawled/crawled by a spider? thanks.



WebmasterWorld Senior Member 10+ Year Member

Msg#: 1852 posted 7:05 am on Dec 9, 2002 (gmt 0)

1) No. The "free" crawl won't happen as a result of your paying for spidering of the index page. Wouldn't make sense, would it? They want you to pay for all of your pages!

2) No. If you don't have a robots.txt you're allowing everybody. No search engine will penalize you for it.


10+ Year Member

Msg#: 1852 posted 7:20 am on Dec 9, 2002 (gmt 0)

Welcome to Webmaster World.

By paying for the index page INK will only spider that page on the 48hr refresh setup. The rest will be indexed in the normal round of updates. I'm not sure if there is any evidence of having a page in the express search submit will speed up the indexing of the rest of your site. If there is someone will come along to correct me.

At one time there was a "problem" with sites that paid the fee for the quick refresh not having the rest of their pages spidered and indexed in the normal INK update but that seems to have gone away now.

Not having a robots.txt file will not hurt you but is a good idea if there are any files or areas that you do not want indexed. Without one basically a spider sees anything there, at the site, is fair game to be crawled and indexed.

If you want to find out more about the robots.txt file click on the site search link at the top of the window and do a search for it.

You'll find that a lot of the questions you might have have already been covered in some detail and you can find those discussions using the site search....plus a lot of other goodies ;)


WebmasterWorld Senior Member tigger us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 1852 posted 8:20 am on Dec 9, 2002 (gmt 0)

I do remember sometime ago that index pages were not ranking well under the paid inclusion program, is anyone still finding this?


10+ Year Member

Msg#: 1852 posted 11:24 pm on Dec 9, 2002 (gmt 0)

Actually for me it is just the opposite. The index page of one site that I used the express deal on is ranked high on multiple KW's while another page, also submited in the express deal, on the same site is buried.


10+ Year Member

Msg#: 1852 posted 12:05 am on Dec 10, 2002 (gmt 0)

For me it's subpages ranking good, index page low, but the competition on the index page keyword is much higher, so I don't think it's because it's the index page

Global Options:
 top home search open messages active posts  

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved