homepage Welcome to WebmasterWorld Guest from 54.196.162.238
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

    
3 Years and still not listed in yahoo
yahoo slurp constantly comes to our site
AprilS




msg:829100
 5:13 pm on Jun 3, 2004 (gmt 0)

I cannot figure this out at all. Our site has been up for over 3 years now and we are still not included in Yahoo!

We cannot be found by using any keywords, our company name, OR just by typing in our domain.

In the past I would watch the logs and see Slurp go through all of our pages (over 2000 pages) and it does it regularly. In fact, I can't think of many days I have not seen yahoo in our logs! Can anyone think of why we have NEVER been listed? I doubt it is our content as our competitors are listed and we do not do anything we are not supposed to do; we run things pretty clean.

What is even more odd is that I've noticed lately that yahoo isn't going to our pages anymore, it just keeps accessing "robots.txt" - which we have - but it is an empty file. What is odd.... Yahoo literally comes to our site ever 3-5 SECONDS!

Any assistance would be greatly appreciated. I can't understand why we are not listed. I doubt we are blacklisted (thought I don't know how to check with Yahoo) - as Yahoo! Slurp comes every 3-5 seconds!

 

helenp




msg:829101
 6:31 pm on Jun 3, 2004 (gmt 0)

If you read around, you will find that you arenīt alone,..
Iīve been trying to figure out since january why slurp only gets my robots.
I only know by now that my site has been blocked for inclusion.

soapystar




msg:829102
 8:03 pm on Jun 3, 2004 (gmt 0)

when you say STILL after 3 years, most of that time was Google serps. As for being clean that doesnt stop you being banned.

AprilS




msg:829103
 8:16 pm on Jun 3, 2004 (gmt 0)

Well, if we were banned, why would yahoo slurp be visiting our site every few seconds for our robots.txt file? Any thought on this? I can't figure out why it would now just be checking our robots.txt file. Does anyone know for sure if Yahoo Slurp "requires" something in the robots.txt file?

Is there a way with Yahoo to figure out if our site has been blocked from inclusion? If so, does anyone have any recommendations on how to contact Yahoo regarding inclusion? I've searched Yahoo and can't find a relavent contact for such a situation.

atlrus




msg:829104
 11:37 pm on Jun 3, 2004 (gmt 0)

And why dont you get rid of the robots.txt and see what's going to happen?

helenp




msg:829105
 7:50 am on Jun 4, 2004 (gmt 0)

>>>And why dont you get rid of the robots.txt and see what's going to happen? >>>

That I done and donīt work..........
if getting only robots = blocked, banned, penalized or any reason for no inclusion.

AprilS




msg:829106
 8:40 am on Jun 4, 2004 (gmt 0)

Well - from what I've read on other posts along with this one, Yahoo is leaving a lot of people hanging.

The only thing I can think of is Yahoo is basically telling the majority of us to pay the $299/$600 annually for inclusion.

Perhaps seeing Yahoo! Slurp in my logs every few seconds is Yahoo's way of getting my interest in Yahoo... who knows!

Has anyone ever written yahoo and received a detailed response?...did it help?

Alternative Future




msg:829107
 9:00 am on Jun 4, 2004 (gmt 0)

>>Well, if we were banned, why would yahoo slurp be visiting our site every few seconds for our robots.txt file?

I think this is the bot following links from other sites to yours, the bot wouldn’t (if this *is* the case) know about any penalty or ban on your site, its just doing its job and trawling the net.

-George

helenp




msg:829108
 10:36 am on Jun 4, 2004 (gmt 0)

>> think this is the bot following links from other sites to yours, the bot wouldn’t (if this *is* the case) know about any penalty or ban on your site, its just doing its job and trawling the net. >>

If it donīt know, why does it only gets the robots? even if you donīt have any robots.txt it only search for the robots, then instead you only get an 404 error.

atlrus




msg:829109
 1:44 pm on Jun 4, 2004 (gmt 0)

Yahoo is just weird like that,
just so far today Yahoo! Surp had crawled my home page 4 times. I am #3 on Overture now, but still not in the top 1000 results of Yahoo for that keyword.

helenp




msg:829110
 1:55 pm on Jun 4, 2004 (gmt 0)

>>Surp had crawled my home page 4 times. I am #3 on Overture now, but still not in the top 1000 results of Yahoo for that keyword.>>

To me sounds like an redirection or algo problem, not exclusion...is your site found if only search for your domain without .com and www?

and if you write your full domain name, does it state ontop did you search for ....similar domain...?

Or does it not show up at all if searching for your full domain? thats the case for sites banned both by yahoo and google.

atlrus




msg:829111
 2:09 pm on Jun 4, 2004 (gmt 0)

:)

Yes, I have done all that. My website appears fine when searched for wwWebmasterWorldebsite.com, mywebsite.com and even when I search for: my web site. And I am #9 on yahoo for a simillar keyword with the url wwWebmasterWorldebsite.com/somethingelse.htm

and linkdomain:wwWebmasterWorldebsite.com brings back 13,700 links

:)

I have no excplanation, maybe I was banned just for a specific keyword or something, I dont know, but as I said so far I have been visited 4 times by Slurp TODAY.

soapystar




msg:829112
 2:55 pm on Jun 4, 2004 (gmt 0)

The only thing I can think of is Yahoo is basically telling the majority of us to pay the $299/$600 annually for inclusion

no, that wont get you into the serps, just the directory. Problem is people still try to find the logic in what yahoo does by assuming they are trying to index the web.

AprilS




msg:829113
 5:33 pm on Jun 4, 2004 (gmt 0)

Thank you everyone for your responses.

So, does anyone know what to do? HOW to get indexed? I've used Yahoo's freed submit MANY times over the past 3 years with no luck at all.

It's sort of embarassing as I see a lot of our competitors listed on Yahoo and quite of few of those sites look like they were designed by grade school kids and make some obvious mistakes that Yahoo states they don't like/want in a website.

This is SOOO frustrating! At this point I would be happy if Yahoo would AT LEAST index our homepage.

soapystar




msg:829114
 5:35 pm on Jun 4, 2004 (gmt 0)

again......for most of the last year it was Google serps..are you in Google?

helenp




msg:829115
 6:46 pm on Jun 4, 2004 (gmt 0)

>>>So, does anyone know what to do? HOW to get indexed?>>>

I would like to know as well, lol :)
Try write to webmasterworldfeedback@yahoo.com to see if you are luckier than me and the rest of the "club".

AprilS




msg:829116
 7:05 pm on Jun 4, 2004 (gmt 0)

soapystar - yes, we have been in Google almost from day one. Google has every single page of our site indexed.

recksul




msg:829117
 7:22 pm on Jun 4, 2004 (gmt 0)

I wrote to webmasterworldfeedback@yahoo.com a few weeks ago about Yahoo never getting past robots.txt. I got quite a prompt reply telling me to take out the robots.txt as Yahoo sometimes has a problem with it. I took it out and the NEXT DAY, Slurp visited almost all my pages. I did get a few indexed, but not all and not the ones I wanted. Now Slurp is back to visiting every day and looking for the robots.txt but going no further. Very frustrating

digitalv




msg:829118
 7:24 pm on Jun 4, 2004 (gmt 0)

The only thing I can think of is Yahoo is basically telling the majority of us to pay the $299/$600 annually for inclusion.

Why wouldn't you want to do this anyway?

AprilS




msg:829119
 6:48 pm on Jun 7, 2004 (gmt 0)

digitalv - well - looks like you are correct!

Here is a follow up. I finally got a response from Yahoo! In a nutshell they said if our site is a "commercial" site then I "must" use the fee-based Yahoo! Express program.

The support person who wrote back saying "According to our records, we have not received a submission request for this URL." - what a joke as I have submitted to yahoo many many times over the past 3 years... and I also see their bot in our logs (looking at robots.txt only though).

It was nice for Yahoo to write back... but I don't think they should hide the fact that you MUST use their fee-based submit for commercial sites (unless I've missed it along the way). They should be straight forward on their submission page and state that paying $299/$600 is the ONLY way to submit a site for indexing. (and if you are a commercial site and never paid to get indexed and listed in results... it sounds like you were lucky and they are enforcing this from now on).

MikeBeverley




msg:829120
 7:14 pm on Jun 9, 2004 (gmt 0)

Oh, you started two thread with the same comment. Ah, well, I'll paste it here too and you can pick where you want to answer it:

Try putting information in your robots.txt file for a start.
Could you answer these questions please:

1. Roughly how many inbound links to you have to your site (not ones from sites you own yourself)?
2. Has your site changed topic/product since it started 3 years ago?
3. Do you provide information/product/service or do you simply re-direct to other sources?
4. Have you ever paid someone else to build you link popularity/get links for you?
5. Were you ever listed in the Inktomi index?
6. Is your main page dynamic?
7. Do you have a sitemap linked to from your main page?
8. How much of your site is in the Google index? (10%, 50%, 90% etc.)

Hopefully with a clearer picture we'll be able to help as much as possible. You could always sticky me your URL if you like and I'll have a look.

Mike

Also, I don't we should be speculating that Yahoo is dropping sites to get people to use Sitematch. My site is highly commercial with hundreds of thousands of product pages and they've never once dropped me. If they were going to do this kind of thing they wouldn't be aiming at the Mom & Pop sites they'd be tackling the big boys but as the big boys are already paying through the nose with Overture and Sitematch they don't need to bother with such underhanded practices.

Mike

your_store




msg:829121
 7:54 pm on Jun 9, 2004 (gmt 0)

They should be straight forward on their submission page and state that paying $299/$600 is the ONLY way to submit a site for indexing.

It seems you're still a bit confused about Y's various programs. The Express submssion your referred to is for the *directory* not *search*. It has nothing to do w/ indexing. The programs are as follows:

Yahoo! Directory [dir.yahoo.com]
- simply a listing of sites
- commercial sites pay $299 for the chance to be included on *one* of the pages
- only bearing on search is the directory *may* be one of the sites the crawler begins

Yahoo! Search [search.yahoo.com]
- powers the actual search results at Yahoo!
- completely independent of the directory
- free [submit.search.yahoo.com] & paid [content.overture.com] inclusion

To increase your chances of being indexed by Yahoo!, you need to aquire links from pages that are heavily crawled by Slurp.

AprilS




msg:829122
 5:21 am on Jun 10, 2004 (gmt 0)

MikeBeverley - Thank you for taking the time to ask your questions. Below are my responses to your questions:

1) We have approximately 220 inbound links.
2) No, we haven't changed topic or products in the past 3 years. We have added some products in that time frame though.
3) We don't redirect to any sources at all. We have thousands of product pages and many many articles (on topic).
4) No, we haven't paid anyone to get links for us.
5) I thought we were..but I can't say 'yes' for sure as its been a while and I can't go back and check...but seeing how inktomi is now yahoo...I would have to assume our site wasn't listed with inktomi.
6) No, it is not dynamic. We update some information on it about once every two weeks though.
7) Yes, we have a site map that is linked from our main page.
8) roughly 95-100% of our site is in the Google index. 3,350 pages.

Also, nice comment regarding sitematch - I admit it was unneeded speculation.

Regarding the "robots.txt" file. You recommended placing information in it. What should I place in the robots.txt file?
User-agent: *
Disallow:

Also, quick note to "your_store" - I am clear about Yahoo's services...I was just relaying what Yahoo stated in their email. I have a feeling that they are very busy and being so... it's easy to send an incorrect response. I wasn't happy with their response so I quickly replied to their email. I will post a follow up when I hear back from Yahoo.

MikeBeverley - I hope the information I provided helps in some way. I'm grateful for any information you can provide!

yaelede




msg:829123
 9:21 am on Jun 10, 2004 (gmt 0)

6. Is your main page dynamic?

How does it affect ranking or possible banning?
Our index page is dynamic, but only because we put a simple search function on this page.
I'm nearly sure that it could be solved in some other way, so that the page remains static.

Could someone confirm the above statement? If so, we will change our index page, because we are banned from the index entirely for a month or so, though we have been in the Yahoo! directory since 2000.
I can't even find our site in the Yahoo! directory search either, even if I use any relevant keyword combination...
..totally confused..

AprilS




msg:829124
 11:29 pm on Jun 11, 2004 (gmt 0)

could someone stickymail me with a phone number for yahoo that I can have them check to see if my site has been penalized in anyway? I'm not getting a response by email (they avoid the question) and the numbers I found online for yahoo have been a deadend as no one was able to help.

Thanks!

markis00




msg:829125
 5:10 pm on Jun 12, 2004 (gmt 0)

Here is the email you should use
webmasterworldfeedback@yahoo.com

I emailed them on this email and they replied the next day; now I'm waiting for another reply (they're checking my sites now)

Don't bother calling them, they're quite stupid on the phone, you'll be on hold for 20+ minutes being charged for the call the whole time, and then when you do talk to someone, they give you the wrong email address.

yaelede




msg:829126
 9:28 pm on Jun 12, 2004 (gmt 0)

I sent an email to the above address two weeks ago, but got no reply at all so far.

So you have two options:
- there are the lucky ones or/and
- there are the selected ones

HarryM




msg:829127
 10:04 pm on Jun 12, 2004 (gmt 0)

Question 5 regarding whether you were ever in Inktomi may be a possible answer to your problems.

My site is Slurped regularly, and yet my site can not be found in Yahoo with a keyword search. I have been told by Yahoo they see no reason for it to be banned, and the only thing I can think of is I had an Inktomi PFI page. I have deactivated the Inktomi page, but no improvement. I have several times emailed Inktomi to remove my account from their database but they never give a direct reply.

markis00




msg:829128
 11:01 pm on Jun 12, 2004 (gmt 0)

Interesting; Yahoo got back to me next day, yaelede

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved