Macguru
msg:222658 | 9:50 pm on Nov 13, 2003 (gmt 0) |
Hi Fizzy, The bot is teasing you. ;) Get more inbound links, else pull the plastic.
|
Fizzy
msg:222659 | 11:04 pm on Nov 13, 2003 (gmt 0) |
Hi Mac, Well I already have over 400 inbound links according to ATW so I'm not sure how many I would need for them to decide to spider the site. I have waited patiently for many months now and have yet to see them go through the site. The pages are shtml so I would have thought that they are spider fodder, it's not even trying to spider them though.
Could this be because ATW is coming in on the inbound links and isn't bothering to actually visit my site to spider it? Could it just be "passing through"?
|
aus_dave
msg:222660 | 11:48 pm on Nov 13, 2003 (gmt 0) |
Fizzy, from what I have seen of ATW it is very slow. On a mid-sized site of mine it is generally about 5-6 months behind in indexing the content when compared to Google. I see lots of requests for robots.txt and then nothing else, similar to what you describe. Some indexing must occur occasionally though, just not very often. I like the ATW interface but my log files tell me it sends me very few visitors.
|
heini
msg:222661 | 11:01 am on Nov 14, 2003 (gmt 0) |
From what I see in my logs the FirstPage crawler has been in charge for index page checking. Deep indexing was from FAST-WebCrawler/3.8.
That one, WebCrawler/3.8./Fresh, really lives up to it's name, it comes daily. Fizzy, there have been several reports lately of sites where only the robots txt gets checked but nothing indexed, even for well linked sites.
Frankly I don't know what the problem is. It might have to do with the behind the scenes working at OV.
We have heard the frontend serving Altavista and ATW has been merged. The bigger question is what about the backend?
|
aus_dave
msg:222662 | 1:12 pm on Nov 14, 2003 (gmt 0) |
heini, sounds like you are getting some nice regular visits :). I contacted the ATW support people by email when I first noticed this robots.txt thing, and they asked for a log file snippet so they could see what was happening. Never heard anything back though.
|
Fizzy
msg:222663 | 10:25 am on Nov 29, 2003 (gmt 0) |
Thanks for the replies everybody. I'll keep watching and see if anything changes, nothing has so far though :(
|
Fizzy
msg:222664 | 11:25 pm on Dec 7, 2003 (gmt 0) |
Hi all, I promised an update if I got it as I was worried about being missed out. FAST-WebCrawler/3.8 - atw-crawler at fast dot no (117 pages and counting) Thanks again to you all for your continued kind and helpful advice.
|
|
