If you read around, you will find that you arenīt alone,..
Iīve been trying to figure out since january why slurp only gets my robots.
I only know by now that my site has been blocked for inclusion.
when you say STILL after 3 years, most of that time was Google serps. As for being clean that doesnt stop you being banned.
Well, if we were banned, why would yahoo slurp be visiting our site every few seconds for our robots.txt file? Any thought on this? I can't figure out why it would now just be checking our robots.txt file. Does anyone know for sure if Yahoo Slurp "requires" something in the robots.txt file?
Is there a way with Yahoo to figure out if our site has been blocked from inclusion? If so, does anyone have any recommendations on how to contact Yahoo regarding inclusion? I've searched Yahoo and can't find a relavent contact for such a situation.
And why dont you get rid of the robots.txt and see what's going to happen?
>>>And why dont you get rid of the robots.txt and see what's going to happen? >>>
That I done and donīt work..........
if getting only robots = blocked, banned, penalized or any reason for no inclusion.
Well - from what I've read on other posts along with this one, Yahoo is leaving a lot of people hanging.
The only thing I can think of is Yahoo is basically telling the majority of us to pay the $299/$600 annually for inclusion.
Perhaps seeing Yahoo! Slurp in my logs every few seconds is Yahoo's way of getting my interest in Yahoo... who knows!
Has anyone ever written yahoo and received a detailed response?...did it help?
>>Well, if we were banned, why would yahoo slurp be visiting our site every few seconds for our robots.txt file?
I think this is the bot following links from other sites to yours, the bot wouldnt (if this *is* the case) know about any penalty or ban on your site, its just doing its job and trawling the net.
>> think this is the bot following links from other sites to yours, the bot wouldnt (if this *is* the case) know about any penalty or ban on your site, its just doing its job and trawling the net. >>
If it donīt know, why does it only gets the robots? even if you donīt have any robots.txt it only search for the robots, then instead you only get an 404 error.
Yahoo is just weird like that,
just so far today Yahoo! Surp had crawled my home page 4 times. I am #3 on Overture now, but still not in the top 1000 results of Yahoo for that keyword.
>>Surp had crawled my home page 4 times. I am #3 on Overture now, but still not in the top 1000 results of Yahoo for that keyword.>>
To me sounds like an redirection or algo problem, not exclusion...is your site found if only search for your domain without .com and www?
and if you write your full domain name, does it state ontop did you search for ....similar domain...?
Or does it not show up at all if searching for your full domain? thats the case for sites banned both by yahoo and google.
Yes, I have done all that. My website appears fine when searched for wwWebmasterWorldebsite.com, mywebsite.com and even when I search for: my web site. And I am #9 on yahoo for a simillar keyword with the url wwWebmasterWorldebsite.com/somethingelse.htm
and linkdomain:wwWebmasterWorldebsite.com brings back 13,700 links
I have no excplanation, maybe I was banned just for a specific keyword or something, I dont know, but as I said so far I have been visited 4 times by Slurp TODAY.
|The only thing I can think of is Yahoo is basically telling the majority of us to pay the $299/$600 annually for inclusion |
no, that wont get you into the serps, just the directory. Problem is people still try to find the logic in what yahoo does by assuming they are trying to index the web.
Thank you everyone for your responses.
So, does anyone know what to do? HOW to get indexed? I've used Yahoo's freed submit MANY times over the past 3 years with no luck at all.
It's sort of embarassing as I see a lot of our competitors listed on Yahoo and quite of few of those sites look like they were designed by grade school kids and make some obvious mistakes that Yahoo states they don't like/want in a website.
This is SOOO frustrating! At this point I would be happy if Yahoo would AT LEAST index our homepage.
again......for most of the last year it was Google serps..are you in Google?
>>>So, does anyone know what to do? HOW to get indexed?>>>
I would like to know as well, lol :)
Try write to email@example.com to see if you are luckier than me and the rest of the "club".
soapystar - yes, we have been in Google almost from day one. Google has every single page of our site indexed.
I wrote to firstname.lastname@example.org a few weeks ago about Yahoo never getting past robots.txt. I got quite a prompt reply telling me to take out the robots.txt as Yahoo sometimes has a problem with it. I took it out and the NEXT DAY, Slurp visited almost all my pages. I did get a few indexed, but not all and not the ones I wanted. Now Slurp is back to visiting every day and looking for the robots.txt but going no further. Very frustrating
|The only thing I can think of is Yahoo is basically telling the majority of us to pay the $299/$600 annually for inclusion. |
Why wouldn't you want to do this anyway?
digitalv - well - looks like you are correct!
Here is a follow up. I finally got a response from Yahoo! In a nutshell they said if our site is a "commercial" site then I "must" use the fee-based Yahoo! Express program.
The support person who wrote back saying "According to our records, we have not received a submission request for this URL." - what a joke as I have submitted to yahoo many many times over the past 3 years... and I also see their bot in our logs (looking at robots.txt only though).
It was nice for Yahoo to write back... but I don't think they should hide the fact that you MUST use their fee-based submit for commercial sites (unless I've missed it along the way). They should be straight forward on their submission page and state that paying $299/$600 is the ONLY way to submit a site for indexing. (and if you are a commercial site and never paid to get indexed and listed in results... it sounds like you were lucky and they are enforcing this from now on).
Oh, you started two thread with the same comment. Ah, well, I'll paste it here too and you can pick where you want to answer it:
|Try putting information in your robots.txt file for a start. |
Could you answer these questions please:
1. Roughly how many inbound links to you have to your site (not ones from sites you own yourself)?
2. Has your site changed topic/product since it started 3 years ago?
3. Do you provide information/product/service or do you simply re-direct to other sources?
4. Have you ever paid someone else to build you link popularity/get links for you?
5. Were you ever listed in the Inktomi index?
6. Is your main page dynamic?
7. Do you have a sitemap linked to from your main page?
8. How much of your site is in the Google index? (10%, 50%, 90% etc.)
Hopefully with a clearer picture we'll be able to help as much as possible. You could always sticky me your URL if you like and I'll have a look.
Also, I don't we should be speculating that Yahoo is dropping sites to get people to use Sitematch. My site is highly commercial with hundreds of thousands of product pages and they've never once dropped me. If they were going to do this kind of thing they wouldn't be aiming at the Mom & Pop sites they'd be tackling the big boys but as the big boys are already paying through the nose with Overture and Sitematch they don't need to bother with such underhanded practices.
|They should be straight forward on their submission page and state that paying $299/$600 is the ONLY way to submit a site for indexing. |
It seems you're still a bit confused about Y's various programs. The Express submssion your referred to is for the *directory* not *search*. It has nothing to do w/ indexing. The programs are as follows:
Yahoo! Directory [dir.yahoo.com]
- simply a listing of sites
- commercial sites pay $299 for the chance to be included on *one* of the pages
- only bearing on search is the directory *may* be one of the sites the crawler begins
Yahoo! Search [search.yahoo.com]
- powers the actual search results at Yahoo!
- completely independent of the directory
- free [submit.search.yahoo.com] & paid [content.overture.com] inclusion
To increase your chances of being indexed by Yahoo!, you need to aquire links from pages that are heavily crawled by Slurp.
MikeBeverley - Thank you for taking the time to ask your questions. Below are my responses to your questions:
1) We have approximately 220 inbound links.
2) No, we haven't changed topic or products in the past 3 years. We have added some products in that time frame though.
3) We don't redirect to any sources at all. We have thousands of product pages and many many articles (on topic).
4) No, we haven't paid anyone to get links for us.
5) I thought we were..but I can't say 'yes' for sure as its been a while and I can't go back and check...but seeing how inktomi is now yahoo...I would have to assume our site wasn't listed with inktomi.
6) No, it is not dynamic. We update some information on it about once every two weeks though.
7) Yes, we have a site map that is linked from our main page.
8) roughly 95-100% of our site is in the Google index. 3,350 pages.
Also, nice comment regarding sitematch - I admit it was unneeded speculation.
Regarding the "robots.txt" file. You recommended placing information in it. What should I place in the robots.txt file?
Also, quick note to "your_store" - I am clear about Yahoo's services...I was just relaying what Yahoo stated in their email. I have a feeling that they are very busy and being so... it's easy to send an incorrect response. I wasn't happy with their response so I quickly replied to their email. I will post a follow up when I hear back from Yahoo.
MikeBeverley - I hope the information I provided helps in some way. I'm grateful for any information you can provide!
|6. Is your main page dynamic? |
How does it affect ranking or possible banning?
Our index page is dynamic, but only because we put a simple search function on this page.
I'm nearly sure that it could be solved in some other way, so that the page remains static.
Could someone confirm the above statement? If so, we will change our index page, because we are banned from the index entirely for a month or so, though we have been in the Yahoo! directory since 2000.
I can't even find our site in the Yahoo! directory search either, even if I use any relevant keyword combination...
could someone stickymail me with a phone number for yahoo that I can have them check to see if my site has been penalized in anyway? I'm not getting a response by email (they avoid the question) and the numbers I found online for yahoo have been a deadend as no one was able to help.
Here is the email you should use
I emailed them on this email and they replied the next day; now I'm waiting for another reply (they're checking my sites now)
Don't bother calling them, they're quite stupid on the phone, you'll be on hold for 20+ minutes being charged for the call the whole time, and then when you do talk to someone, they give you the wrong email address.
I sent an email to the above address two weeks ago, but got no reply at all so far.
So you have two options:
- there are the lucky ones or/and
- there are the selected ones
Question 5 regarding whether you were ever in Inktomi may be a possible answer to your problems.
My site is Slurped regularly, and yet my site can not be found in Yahoo with a keyword search. I have been told by Yahoo they see no reason for it to be banned, and the only thing I can think of is I had an Inktomi PFI page. I have deactivated the Inktomi page, but no improvement. I have several times emailed Inktomi to remove my account from their database but they never give a direct reply.
Interesting; Yahoo got back to me next day, yaelede