Welcome to WebmasterWorld Guest from 54.196.245.74

Forum Moderators: open

Message Too Old, No Replies

How do you get your site on ATW?

I'm new to this game!

     

rainer

10:37 pm on Sep 19, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


Could someone please tell me how to get your site on ATW?
I submitted my site to them via my hosting provider (They submitted my site to 8 search engines)

I have been having visits to my website from Fast PartnerSite Crawler, who have been looking for robots.txt
What is robots.txt, and what should I do?

10:49 pm on Sept 19, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 27, 2002
posts:1187
votes: 0


Hi Rainer,
Welcome to Webmasterworld.

All The Web also has a free submission here [alltheweb.com]. It takes time but it works.

The robots.txt file is explained in a lot of threads here. Here [webmasterworld.com]

also check out robotstxt.org [robotstxt.org]
and
Searchengineworld [searchengineworld.com].

in short....
To exclude all robots from the entire server
User-agent: *
Disallow: /

To allow all robots complete access
User-agent: *
Disallow:

Or create an empty "/robots.txt" file.

There is lots of great community here. Look around, ask questions. There is always someone hanging out.

jb

10:52 am on Sept 23, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 25, 2003
posts:128
votes: 0


It does take time.
I submitted a site over 3 months ago and I'm still waiting.
I submitted to Google at the same time and the site was crawled within a week.

Ove

10:57 am on Sept 23, 2003 (gmt 0)

Senior Member from SE 

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 24, 2001
posts:786
votes: 0


Hello rainer and welcome to WebmasterWorld.

It take some times i ATW a much longer than before, there are no simple rules, just links to your site from other sites good content and a lot of patient.

/Ove

[edited by: Ove at 11:06 am (utc) on Sep. 23, 2003]

11:05 am on Sept 23, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 21, 2001
posts:2488
votes: 0


>I submitted my site to them via my hosting provider (They submitted my site to 8 search engines)

Unless you are specifically paying for SEO services from your Host, then i would'nt heed to much towards that approach.

If you want a quick inclusion then the PFI is a 48 hour inclusion via 'positiontech' obviously at cost.

Otherwise as the others have said, get reciprocated links in your industry, and get on the crawl/index cycle.

6:29 pm on Oct 7, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 17, 2003
posts:169
votes: 0


How do you check the pages from your site that ATW has indexed?

I only know "+link.all:www.domain.com -site:domain.com" which checks for external links to your site.

6:31 pm on Oct 7, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 17, 2003
posts:169
votes: 0


caine,

How do you get ATW to index more pages without PFI? I have some pages indexed but would like to have a lot more in there.

6:21 pm on Oct 8, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 25, 2001
posts:141
votes: 0


benc007, I sometimes have a hard time getting all my subpages of a domain name indexed, and when they are indexed they are burried. Depending on how competitve the industry you're in you're going to see a lot of this. Index pages are usually more important than subpages to search engines in my experience - then again if your industry is not competitive at all you might have no trouble getting your subpages up and ranking well.

To get your subpages indexed, do what you would do with any index page - optimize it the best you can, and try to get some incoming links pointed at that subpaged with relevant anchor text.

But yeah, for the most part, I'm in the same boat as you are. All you can do is keep trying and be patient.

Ove

7:11 pm on Oct 8, 2003 (gmt 0)

Senior Member from SE 

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 24, 2001
posts:786
votes: 0


There should not be any problems to get the subpages in the index as long you have some links to that pages, if your indexpage is there the crawler will get get there, and if you not rank good for it try some other seo technices for the subpages and you will do fine, but these days i dont know for sure which crawler you should look for it happens alot to ATW right now :-)

/Ove

5:44 am on Oct 12, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 17, 2003
posts:169
votes: 0


I noticed one of the competitors have 20,000 pages indexed in ATW while others only have 500 pages. My client has about 1,000 pages but only 50 pages are on ATW. Any suggestions? I have external links pointing to sub pages.

Does ATW have a paid option where I can give them URLs to spider?

9:44 am on Oct 12, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 18, 2003
posts:61
votes: 0


Does ATW have a paid option where I can give them URLs to spider?

ATW do offer PFI at $34 per url, but normally the bot would come and spider just the page that you have submitted. Many months will pass until he/she decides to spider and index the rest of the site.

At least this is what I have been observing for 6 months.

IMHO

5:35 am on Oct 16, 2003 (gmt 0)

Junior Member

joined:Nov 3, 2002
posts:131
votes: 0


My experience with atw is you pay then they spider the one page then eventually drop your site hoping you pay again....?

Don't even submit to them and they come crawling big time.

8:16 am on Nov 18, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 17, 2003
posts:169
votes: 0


CoolCreep,

How many pages do you manage to get into ATW without paying? How long did this take?

12:58 pm on Nov 18, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 15, 2003
posts:329
votes: 0


Ive never used PFI and alltheweb only index's the page that submitted free, although im very happy with my #1 position for my chosen keywords. Really the page i submitted is the one i want people to enter on....and its #1 so thats all that matters to me.
1:05 pm on Nov 18, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 15, 2003
posts:329
votes: 0


I forgot to mention that Fast has come for a visit almost every day for the last week.
4:30 pm on Nov 18, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 17, 2003
posts:169
votes: 0


What command do you use to find the pages that ATW / Fast has indexed?
5:23 pm on Nov 18, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 15, 2003
posts:329
votes: 0


type the full address into the search box, if its in the index it gives you info, if not it returns.
No entries for www.yourdomain.com/page.htm in our index.
1:58 pm on Nov 19, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 8, 2001
posts:665
votes: 0


Just to add to the good advice from Caine (and from all), it may be worth a check to see that you ISP is NOT submitting your site URL literally every day, or weekly etc.

Whilst I have no proof, I am sure that ATW will have some kind of penalty for site URL's submitted repeatedly over a short period of time.

It is a must to invest in some dedicated SEO and wider e-marketing to promote your site effectively. I have only every used ATW's 'free' submission, and whilst it has been a bit patchy in listing the pages I want, they do get in - even if they flunctuate quite a lot.

11:56 pm on Nov 19, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 17, 2003
posts:169
votes: 0


I just checked ATW and our site is listed. I noticed our subdomain is not though. Should I submit this manually?
9:28 am on Nov 20, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 24, 2003
posts:86
votes: 0


if i dont create an robots.txt, all the spiders will crawl my site?
2:41 pm on Nov 20, 2003 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
posts:2531
votes: 1


>if i dont create an robots.txt, all the spiders will crawl my site?

In a perfect world, yes.

However you should always put a robots on your server, even if it's empty. All good spiders will ask for it and obey it too.

11:18 am on Nov 28, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 25, 2003
posts:128
votes: 0


Hold the front page!

My site had its first crawl from atw last night.
5 months and 3 weeks since the site was put up.
I have been submitting at 30 day intervals during that time.
It took about 20 pages

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members