Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
I've had a good read around this forum but can't see what I'm missing. I have some good incoming links, including DMOZ, meta tags and titles in place.
Can there be something obvious that I'm missing?
joined:Nov 8, 2002
If your robots.txt is not set up for any particular purpose, you might would be better off deleting it.
[edited by: WebManager at 6:51 pm (utc) on Nov. 11, 2002]
joined:Nov 8, 2002
Unfortunately it may be that the site isn't considered relevant / important enough due to their algo. I had a site that was visited again and again by Google - and it sometimes showed up in their minty fresh updates for a week or so - and then disappeared again.
It was highly relevant to a human reader, but I had to make some basic changes to the index page to convince the spider.
Are your titles, keywords and content such that what your site claims to be about is clear to a semi-intelligent spider?
In some cases I get a prompt for a cert in Linux/Mozilla on the 404 pages of raq's, maybe just that an image is being called from the server root. If you don't have access to the server then it may be wise to upload a valid robots.txt and avoid that particular problem.
On a sidenote Fast seem to be very responsive to email, if you have a problem after this I would mail them, I'd be surprised if you didn't get a helpful reply.
<<After looking into this, we found that your robots.txt file is set to:
The asterisk in the User-agent area effectively blocks the crawlers from
indexing any crawled information. A suggestion would be to replace the
asterisk with the user-agent name or names of the robots you are trying
to block. As long as FAST is not on your robots.txt user-agent list,
then we would be able to index the information we crawl.
We hope that is helpful to you.>>
Now I believe that shouldn't be the case...or am I missing something?
This should allow all robots:
This should disallow all robots:
I recommend writing them back for confirmation. I think they erred in their response.