How do you get your site on ATW?

I'm new to this game!

[rainer]

Could someone please tell me how to get your site on ATW?
I submitted my site to them via my hosting provider (They submitted my site to 8 search engines)

I have been having visits to my website from Fast PartnerSite Crawler, who have been looking for robots.txt
What is robots.txt, and what should I do?

[jbinbpt]

Hi Rainer,
Welcome to Webmasterworld.

All The Web also has a free submission here [alltheweb.com]. It takes time but it works.

The robots.txt file is explained in a lot of threads here. Here [webmasterworld.com]

also check out robotstxt.org [robotstxt.org]
and
Searchengineworld [searchengineworld.com].

in short....
To exclude all robots from the entire server
User-agent: *
Disallow: /

To allow all robots complete access
User-agent: *
Disallow:

Or create an empty "/robots.txt" file.

There is lots of great community here. Look around, ask questions. There is always someone hanging out.

jb

[jon80]

It does take time.
I submitted a site over 3 months ago and I'm still waiting.
I submitted to Google at the same time and the site was crawled within a week.

[Ove]

Hello rainer and welcome to WebmasterWorld.

It take some times i ATW a much longer than before, there are no simple rules, just links to your site from other sites good content and a lot of patient.

/Ove

[edited by: Ove at 11:06 am (utc) on Sep. 23, 2003]

[caine]

>I submitted my site to them via my hosting provider (They submitted my site to 8 search engines)

Unless you are specifically paying for SEO services from your Host, then i would'nt heed to much towards that approach.

If you want a quick inclusion then the PFI is a 48 hour inclusion via 'positiontech' obviously at cost.

Otherwise as the others have said, get reciprocated links in your industry, and get on the crawl/index cycle.

[benc007]

How do you check the pages from your site that ATW has indexed?

I only know "+link.all:www.domain.com -site:domain.com" which checks for external links to your site.

[benc007]

caine,

How do you get ATW to index more pages without PFI? I have some pages indexed but would like to have a lot more in there.

[Michael Weir]

benc007, I sometimes have a hard time getting all my subpages of a domain name indexed, and when they are indexed they are burried. Depending on how competitve the industry you're in you're going to see a lot of this. Index pages are usually more important than subpages to search engines in my experience - then again if your industry is not competitive at all you might have no trouble getting your subpages up and ranking well.

To get your subpages indexed, do what you would do with any index page - optimize it the best you can, and try to get some incoming links pointed at that subpaged with relevant anchor text.

But yeah, for the most part, I'm in the same boat as you are. All you can do is keep trying and be patient.

[Ove]

There should not be any problems to get the subpages in the index as long you have some links to that pages, if your indexpage is there the crawler will get get there, and if you not rank good for it try some other seo technices for the subpages and you will do fine, but these days i dont know for sure which crawler you should look for it happens alot to ATW right now :-)

/Ove

[benc007]

I noticed one of the competitors have 20,000 pages indexed in ATW while others only have 500 pages. My client has about 1,000 pages but only 50 pages are on ATW. Any suggestions? I have external links pointing to sub pages.

Does ATW have a paid option where I can give them URLs to spider?

[d_fused]

Does ATW have a paid option where I can give them URLs to spider?

ATW do offer PFI at $34 per url, but normally the bot would come and spider just the page that you have submitted. Many months will pass until he/she decides to spider and index the rest of the site.

At least this is what I have been observing for 6 months.

IMHO

[coolcreep]

My experience with atw is you pay then they spider the one page then eventually drop your site hoping you pay again....?

Don't even submit to them and they come crawling big time.

[benc007]

CoolCreep,

How many pages do you manage to get into ATW without paying? How long did this take?

[thehittmann]

Ive never used PFI and alltheweb only index's the page that submitted free, although im very happy with my #1 position for my chosen keywords. Really the page i submitted is the one i want people to enter on....and its #1 so thats all that matters to me.

[thehittmann]

I forgot to mention that Fast has come for a visit almost every day for the last week.

[benc007]

What command do you use to find the pages that ATW / Fast has indexed?

[thehittmann]

type the full address into the search box, if its in the index it gives you info, if not it returns.
No entries for www.yourdomain.com/page.htm in our index.

[markd]

Just to add to the good advice from Caine (and from all), it may be worth a check to see that you ISP is NOT submitting your site URL literally every day, or weekly etc.

Whilst I have no proof, I am sure that ATW will have some kind of penalty for site URL's submitted repeatedly over a short period of time.

It is a must to invest in some dedicated SEO and wider e-marketing to promote your site effectively. I have only every used ATW's 'free' submission, and whilst it has been a bit patchy in listing the pages I want, they do get in - even if they flunctuate quite a lot.

[benc007]

I just checked ATW and our site is listed. I noticed our subdomain is not though. Should I submit this manually?

[Josecito]

if i dont create an robots.txt, all the spiders will crawl my site?

[Rumbas]

>if i dont create an robots.txt, all the spiders will crawl my site?

In a perfect world, yes.

However you should always put a robots on your server, even if it's empty. All good spiders will ask for it and obey it too.

[jon80]

Hold the front page!

My site had its first crawl from atw last night.
5 months and 3 weeks since the site was put up.
I have been submitting at 30 day intervals during that time.
It took about 20 pages