|Can't get listed in Fast/Lycos|
Can't get my site listed in Fast/ATW/Lycos
| 8:01 am on Aug 2, 2002 (gmt 0)|
For some reason I can't get my hobby site listed on Fast/ATW/Lycos and was wondering if there are any bugs that would get it rejected? I don't mean an html check, I mean weird stuff that might keep a site out like "analysis" or "sextant" being taken as adult content, etc.
I've tried submitting for three years now and only gotten into Fast for a couple of months and then one day the site was just gone from the listing. I've tried adjusting the meta to avoid what might be key word spam. I've tried both short /topic.html and full internal links. I've tried submitting both h...mysite.com/ and h...mysite.com/index.html, nothing works.
I've joined Lycos and submitted as a member which is supposed to get my home page listed, it didn't. When I was able to get past the automated help response which suggested I pay for the listing, I was told there was nothing wrong with the site and I should submit again which again didn't work. I've tried submitting through Lycos in other countries, I've added a Fast, an All the Web and a Lycos search box to internal pages, I'm in Hotbot through Ink and in the Lycos Directory through DMOZ but of course their search doesn't find that.
I've even considered if it's the hyphen in the url and title which does have an affect on boolian searches where a (-) means don't find. I do have about a 100 pages that are identical except for number on the charts I've tried putting robot exclusions on these if that could be a cause of rejection. The site is hosted on a "free server" which I pay to be free of their ads and for extra bandwidth and I own my domain name. I have wondered if the free domains out there for my site could be considered spamming, but I only submit the paid domain.
One last thing. Each time I've gotten through and someone from Lycos or Fast looked over the situation, pages linking to my site that would come up in searches for my site would drop out of the listing. I had comet and search hippo and a directory linking direct to my page within a frame and those listing went away after I inquired about mine, is it my site or what?
Comments, suggestions, jokes?
[edited by: heini at 8:55 am (utc) on Aug. 2, 2002]
[edit reason] we don't do site reviews - thanks [/edit]
| 8:29 am on Aug 2, 2002 (gmt 0)|
Robert, it sounds very odd to me. The site is listed on other engines - inktomi, google, av etc.?
It is also listed in ODP, Yahoo etc.?
Have you tried putting Bretts Sim Spider [searchengineworld.com] on it?
What about your logs, does they show the Fast crawler visiting? If so, how many pages does it take - all of them? None?
Adult words shouldn't keep Fast from spidering your site. Not being listed for 3 years sounds very strange and I'd say anything's possible.
Com' on, anybody have a clue here?
| 5:44 pm on Aug 2, 2002 (gmt 0)|
The Sim Spider came up 200 no errors returned, but for some reason has the meta keywords box blank.
The site is listed pretty much everywhere else that matters, including Google, Yahoo, DMOZ, Ink, etc.
The only other search I can't get on is Scrub the Web and they told me this . . .
"STW randomly indexes submitted sites at this time which probably explains why you have not made it in the index. Even I have two personal sites and have only been lucky enough to get one indexed. However a new Scrubby bot will be released within the next couple of months. In fact our entire robot and search engine has been undergoing a complete rewrite. Right now we have no control over what Scrubby indexes and doesn't index. Scrubby just runs and runs and runs... We should see much improvement over the new system. That's about all I know at this time."
| 5:51 pm on Aug 2, 2002 (gmt 0)|
I am having the same problem with a couple of my sites. I can get it in Inktomi, dmoz, google, whatever. No fast. Lycos mod spider comes around each dmoz listing but never comes back. It makes me have very little faith in FAST's ability to have relevent searches. I think they are too busy looking at pay to index rather than quality of their index. People around here say they have no problems. I have talked to many people who can't get large nice relevent sites in.
| 9:59 pm on Aug 2, 2002 (gmt 0)|
I have this problem too.
I have a theory, but it is only a theory as to what my particular problem is:
Years ago I when I started on a free host, I bought a domain and paid for forwarding by the registrar to my website. Not really knowing what I was doing, I started using this domain when asking for links back. (I was considering moving to a new host so it seemed like a prudent thing at the time.)
So, my luck is that ATW spiders that forwarding domain in first and considers it the site, while the real site on its present host cannot get listed. That is my theory anyway.
Robert could something like that be effecting you?
| 7:50 am on Aug 3, 2002 (gmt 0)|
I never used a redirect so I don't think that's the problem. My "free" host I pay did make sub-domains they released to "promote" the site to search engines and I occasionally come across one which goes nuts with shortened links if someone tries to use it. Trouble is Lycos/Fast/ATW doesn't find either, if they did prefer the sub-domain I'd just go back to full internal links as I don't notice any speed increase from this "tweak" anyway.
No, there is something the Fast spider doesn't like about my site and I'd sure like to know what it is. The only other thing I can come up with is perhaps I've submitted the site too often from my email/dial-up IP address, maybe it would help if a few others would be kind enough to submit the site to Fast, otherwise it's really subtle or just plain nuts which it's making me. :-(
| 10:06 am on Aug 3, 2002 (gmt 0)|
Robert, FAST is a free spidering engine. They have a db of 2.1 Bill pages, of which far less than 1% is paid for.
Submissions are not neccessary, and in you case rather useless.
Having a dmoz listing and other good inbound links is the best way to get picked up by FAST. If you still do not get included over a long period of time I'm afraid FAST really has some problem with your site.
Sorry to say that, but if you are keen on getting listed you could always pay...
Another question: Did you try and look if any sites from your IP are listed in ATW? You could use the IP search under the advanced options.
| 2:09 pm on Aug 3, 2002 (gmt 0)|
Just a long shot...
Did you always use this much bold text ? The pages might seem statistically strange compared to an average page, due to the low ratio regular-to-bold text..
It is conceivable that Fast would use a spam-filter on this if bold text were to help rankings..
| 1:57 am on Aug 4, 2002 (gmt 0)|
Hmmm, never considered too much bold/strong text could make a difference. Guess I could switch over to font 2, wonder if it makes a difference if the submitted page is one way and the rest of the site is the other? Seems more of a choice an editor should be making than a spider.
| 6:17 am on Aug 14, 2002 (gmt 0)|
Well my site still isn't listed on Fast/ATW/Lycos. I tried the suggestion not to use all Bold/Strong text which I feel necessary to be readable over the background graphic. I tried a font +1 which didn't look as good, but at least was a change, still no joy.
I'm not looking for a site review, don't care what people think of it, it's just a hobby site, but it is making me nuts trying to figure out what keeps it out of Lycos/ATW/Fast and still leaves it acceptable to all other major and most minor search and directories. Some have suggested Fast/ATW/Lycos consider more than three of any key word to be spam. As another minor engine I can't get in says more than seven times is spam I trimmed them down to six or fewer appearences in various forms of phrases, again no success.
A red herring might be all the mistakes on the site the html validator finds, they are mostly in code from outside sources such as my news feed, affiliate links and web stats provider (funny how that works).
Where's all the Fast experts? Someone's gotta have a clue for me!
| 7:56 am on Aug 14, 2002 (gmt 0)|
Robert, has Fast spidered your site inbetween? If not it's not surprising you're still not listed.
You should however try to correct those and get your pages validating. Not sure if this prevents the site from getting listed, though.
At this point Fast appears to be somewhat erratic with getting at new sites which they have not indexed yet.
In general, as frustrating as it may sound, Fast like any other engine is completely free as to what they index or not. No engine indexes all the web, they all leave out significant portions.
If you're really that keen on getting listed you might consider paying for just one page and see what happens.
| 2:12 pm on Aug 14, 2002 (gmt 0)|
I've had the same problem of AllTheWeb not listing several sites. No spam. Other search engines and directories pick the sites up and list them, and with good ranking. I'm suspecting the server I'm on has been banned because someone else on the server did some spamming. But I don't know.
| 5:23 am on Aug 15, 2002 (gmt 0)|
Paying for a listing in Fast/ATW and getting in won't tell me what's currently keeping my site out so I can fix the problem. Besides I can't see paying for what I am entitled to free for being a member of Lycos and accepting their advertising, that alone is supposed to almost guarantee one page listing. Even the submitt form says at the end the site will be added not to mention my hosting a Lycos, Fast and ATW search box on different interior pages. I can't even write and ask anymore because the last time they removed some of the site listings linking to my site. This is more like an itch I can't scratch than a real need to be in still I'd rather be in than out.
The site is a legitimate free resource it should be listed free in anything calling itself a general search engine and other than Lycos/Fast it is. I can't understand it, sorry.
| 6:17 pm on Aug 26, 2002 (gmt 0)|
Going on the assumption there is something about my home page that offends the Fast/ATW/Lycos spider, is it possible to make a simplified version and submit it only to Fast/ATW/Lycos as www.mysite.com/index2.html and if so what would the potential draw backs be?