Page is a not externally linkable
-- Deprecated - Altavista, Alltheweb.com
---- Every week FAST's crawler fetches robots.txt & leaves without crawling
luma - 1:41 am on Jul 11, 2002 (gmt 0)Thread source:: http://www.webmasterworld.com/altavista/541.htm
Hi jdMorgan, thanks for your answer and thanks for everyone who had been checking.
I do use .htaccess for some 301s but haven't figured out blocking UAs.
|Your robots.txt looks OK to me... The only thing I see is that your three Disallows for "Microsoft URL Control" are not likely to work - those user-agents likely won't check robots.txt at all. You should probably block these in .htaccess (for Apache server) instead. |
Yes other pages linking to me are in.
|Does Fast index any of the sites that link to your site? If so, they should pick up your site quickly. |
|In your log files, what server code does your server return when Fast requests robots.txt? |
No. But I only started adding real content and getting links to it a couple of months ago.
|Have you recently moved your site? |
I am sure I submitted a page or two a couple of months ago (free submit) and did so again a couple of days ago. I might be wrong cause I don't keep notes...
|Have you tried Fast's "Submit a Site" process? |
I am using Linux myself and checked again but everything seems right.
|Another "picky" thing about robots.txt is that is a Unix-format file; |
Thanks for your help. You see, I think I double-checked everything and really can't find anything. :(
|So, good question - Anyone else? |
Well, I just read thread Kudos to AllTheWeb - Customer Service [webmasterworld.com], so maybe there's hope after all. I used the AllTheWeb.com: Send Feedback to FAST [alltheweb.com] form a couple of days ago. But I also wanted to be sure that there's not a general problem (that could affect other search engines as well).
Guess I will just have to wait and hope that Google will never start ignoring me. It's just that I don't want to put all eggs in one basket ...
Brought to you by WebmasterWorld: http://www.webmasterworld.com