ATW! Make me smile and make me cry ...

[jamesyap]

Today, I saw ATW bot reaching my site. Was very excited and happy, but it only takes 20 files and sadly those files are NOT IMPORTANT at all pages. ATW bot come to my index page, it didn't crawl the link from the top to bottom. Instead it take the 1st link, get into it and get another 3 pages. It also get most of the links at the end of the root page instead of those important ones in the very beginning. Huh.

OHHH, ATW PLEASE COME BACK TO ME! I AM YOURS!

:)

[SEGuru2]

That does sound kinda screwy for a crawler in my opinion. Sounds kinda like a crap shoot...cover your bases and hope you make your point! :)

[jamesyap]

The bot still hasn't come back for the last 12 hours! huh ... come on baby, come on baby!

[Rumbas]

I bet it will come back soon. It has a_lot of sites to run through. Don't worry too much - make a few pages while you wait :)

[jamesyap]

Hehe, just have a few link exchange while I wait ..

too bad wake up today buy Fast didn't pay a visit the whole night? :(

I think they are smart, they help me to crawl pages like - contact us, about us, links, terms of use, privacy policy ... so that I would need to pay to index my real contents. :) (just joking) Just wish they would index the complete site for the next update, instead of the next next update.

[jamesyap]

It looks like ATW has just update their database. Most of the pages the crawler get 2 days ago appeared in th database! Wow, amazing on the speed of crawl -> database = 2 days. While google has a freshbot which is quite fast, but its real indexing happens only once a month and crawl -> database = 1 month!

Looks like I will need to wait for the next update before it come back to take the more valuable pages.

Question:
What is ATW update cycle? 2 weeks?

[rfgdxm1]

I must say I've got the right SEO touch with ATW looking at the current results. If only Google were this easy. (rfgdxm grumbles about Google's excessive weighting of inbound link anchor text.)

[Rumbas]

>What is ATW update cycle? 2 weeks?

Happens on a ongoing basis - smaller chunks at a time, around 25% I think. Sometimes we see a major update though.

Been a while since I really dug into it though. Anybody with more specific numbers?

[jamesyap]

Google's excessive weighting of inbound link anchor text

Hmm, this might be a good news or bad news. Since we know 'how it works' and after we do some 'keyword research', we have the ability to raise the ranking by using the correct anchor text. :)

But it is quite frustrated that google don't really care on the theme where the link is file. Google should check the title of the page with the link and compare it to its link to make them more relevant.

For ATW, sometimes I do see funny results appearing #2 and #3. It is better to be irrevalant so that people will still checking other sites below until they reach mine. haha. What's good on ATW is we don't need to wait so long to see the results appearing.

[heini]

>It also get most of the links at the end of the root page instead of those important ones in the very beginning.

Same story with a site here. Had initially PFI'ed a couple of selling pages from the site. Now the main crawler has hit the site, took links from a secondary bottom navbar.
An interesting tidbit here is the free pages already appear in the db, but without the usual title and snippet.
Instead ATW gives the filename as title, and the linktext as snippet.

[jamesyap]

ATW takes everything from the root directory /files.php and ignore those in subdir /subdir/files.php in my case

[Rumbas]

That's strange jamesyap. I have plenty of sites when pages in subdirectories rank just fine. I assume that you have links to those pages?

One reason could be the clustering/collapsing [webmasterworld.com] of pages from the same site that we've seen FAST do.

[jamesyap]

Rumbas,

My site is new, so ATW come step by step, the 1st update takes only the main page, 2 weeks later it follows all the links in the main page but in the main directory, I believe on the next update, it will reach all.

For the root index lost page bug, it happend to one of my site and friendly sole from ATW has help me to solve it.