homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

How do you get your site on ATW?
I'm new to this game!

 10:37 pm on Sep 19, 2003 (gmt 0)

Could someone please tell me how to get your site on ATW?
I submitted my site to them via my hosting provider (They submitted my site to 8 search engines)

I have been having visits to my website from Fast PartnerSite Crawler, who have been looking for robots.txt
What is robots.txt, and what should I do?



 10:49 pm on Sep 19, 2003 (gmt 0)

Hi Rainer,
Welcome to Webmasterworld.

All The Web also has a free submission here [alltheweb.com]. It takes time but it works.

The robots.txt file is explained in a lot of threads here. Here [webmasterworld.com]

also check out robotstxt.org [robotstxt.org]
Searchengineworld [searchengineworld.com].

in short....
To exclude all robots from the entire server
User-agent: *
Disallow: /

To allow all robots complete access
User-agent: *

Or create an empty "/robots.txt" file.

There is lots of great community here. Look around, ask questions. There is always someone hanging out.



 10:52 am on Sep 23, 2003 (gmt 0)

It does take time.
I submitted a site over 3 months ago and I'm still waiting.
I submitted to Google at the same time and the site was crawled within a week.


 10:57 am on Sep 23, 2003 (gmt 0)

Hello rainer and welcome to WebmasterWorld.

It take some times i ATW a much longer than before, there are no simple rules, just links to your site from other sites good content and a lot of patient.


[edited by: Ove at 11:06 am (utc) on Sep. 23, 2003]


 11:05 am on Sep 23, 2003 (gmt 0)

>I submitted my site to them via my hosting provider (They submitted my site to 8 search engines)

Unless you are specifically paying for SEO services from your Host, then i would'nt heed to much towards that approach.

If you want a quick inclusion then the PFI is a 48 hour inclusion via 'positiontech' obviously at cost.

Otherwise as the others have said, get reciprocated links in your industry, and get on the crawl/index cycle.


 6:29 pm on Oct 7, 2003 (gmt 0)

How do you check the pages from your site that ATW has indexed?

I only know "+link.all:www.domain.com -site:domain.com" which checks for external links to your site.


 6:31 pm on Oct 7, 2003 (gmt 0)


How do you get ATW to index more pages without PFI? I have some pages indexed but would like to have a lot more in there.

Michael Weir

 6:21 pm on Oct 8, 2003 (gmt 0)

benc007, I sometimes have a hard time getting all my subpages of a domain name indexed, and when they are indexed they are burried. Depending on how competitve the industry you're in you're going to see a lot of this. Index pages are usually more important than subpages to search engines in my experience - then again if your industry is not competitive at all you might have no trouble getting your subpages up and ranking well.

To get your subpages indexed, do what you would do with any index page - optimize it the best you can, and try to get some incoming links pointed at that subpaged with relevant anchor text.

But yeah, for the most part, I'm in the same boat as you are. All you can do is keep trying and be patient.


 7:11 pm on Oct 8, 2003 (gmt 0)

There should not be any problems to get the subpages in the index as long you have some links to that pages, if your indexpage is there the crawler will get get there, and if you not rank good for it try some other seo technices for the subpages and you will do fine, but these days i dont know for sure which crawler you should look for it happens alot to ATW right now :-)



 5:44 am on Oct 12, 2003 (gmt 0)

I noticed one of the competitors have 20,000 pages indexed in ATW while others only have 500 pages. My client has about 1,000 pages but only 50 pages are on ATW. Any suggestions? I have external links pointing to sub pages.

Does ATW have a paid option where I can give them URLs to spider?


 9:44 am on Oct 12, 2003 (gmt 0)

Does ATW have a paid option where I can give them URLs to spider?

ATW do offer PFI at $34 per url, but normally the bot would come and spider just the page that you have submitted. Many months will pass until he/she decides to spider and index the rest of the site.

At least this is what I have been observing for 6 months.



 5:35 am on Oct 16, 2003 (gmt 0)

My experience with atw is you pay then they spider the one page then eventually drop your site hoping you pay again....?

Don't even submit to them and they come crawling big time.


 8:16 am on Nov 18, 2003 (gmt 0)


How many pages do you manage to get into ATW without paying? How long did this take?


 12:58 pm on Nov 18, 2003 (gmt 0)

Ive never used PFI and alltheweb only index's the page that submitted free, although im very happy with my #1 position for my chosen keywords. Really the page i submitted is the one i want people to enter on....and its #1 so thats all that matters to me.


 1:05 pm on Nov 18, 2003 (gmt 0)

I forgot to mention that Fast has come for a visit almost every day for the last week.


 4:30 pm on Nov 18, 2003 (gmt 0)

What command do you use to find the pages that ATW / Fast has indexed?


 5:23 pm on Nov 18, 2003 (gmt 0)

type the full address into the search box, if its in the index it gives you info, if not it returns.
No entries for www.yourdomain.com/page.htm in our index.


 1:58 pm on Nov 19, 2003 (gmt 0)

Just to add to the good advice from Caine (and from all), it may be worth a check to see that you ISP is NOT submitting your site URL literally every day, or weekly etc.

Whilst I have no proof, I am sure that ATW will have some kind of penalty for site URL's submitted repeatedly over a short period of time.

It is a must to invest in some dedicated SEO and wider e-marketing to promote your site effectively. I have only every used ATW's 'free' submission, and whilst it has been a bit patchy in listing the pages I want, they do get in - even if they flunctuate quite a lot.


 11:56 pm on Nov 19, 2003 (gmt 0)

I just checked ATW and our site is listed. I noticed our subdomain is not though. Should I submit this manually?


 9:28 am on Nov 20, 2003 (gmt 0)

if i dont create an robots.txt, all the spiders will crawl my site?


 2:41 pm on Nov 20, 2003 (gmt 0)

>if i dont create an robots.txt, all the spiders will crawl my site?

In a perfect world, yes.

However you should always put a robots on your server, even if it's empty. All good spiders will ask for it and obey it too.


 11:18 am on Nov 28, 2003 (gmt 0)

Hold the front page!

My site had its first crawl from atw last night.
5 months and 3 weeks since the site was put up.
I have been submitting at 30 day intervals during that time.
It took about 20 pages

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved