homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

Frequency of yahoo crawler
I think we have hit a jackpot :)

10+ Year Member

Msg#: 2476 posted 1:15 am on Jul 4, 2004 (gmt 0)

Yahoo crawler used to come to our site once in few days.. we recently started second round of link development and we got about 20-30 new links in past week.. we had about 50 links earlier.
Since yesterday yahoo crawler is visiting the home page every 3-4 hours.. does that mean we have crossed some link threshold which qualifies us as a better site (compared to before)... we are getting hits from this crawler:

Although this url (HTTP://WWW.ALLTHEWEB.COM/HELP/WEBMASTER/CRAWLER) does not exists, but this is what we are getting in crawler identification.



10+ Year Member

Msg#: 2476 posted 2:49 am on Jul 7, 2004 (gmt 0)

Yes, I think you are so lucky and you did so well! Opposite to your case, Yahoo! crawler haven't come to my site for a long time, even though I update my site frequently.
I don't know how to attract Yahoo! crawler come to get my updated changing? My site get the 3th position on Google, but very low position on Yahoo!
Need your help! Thanks a lot.


10+ Year Member

Msg#: 2476 posted 5:09 am on Jul 7, 2004 (gmt 0)

The "formerwebcrawler" is not the web search crawler. Yahoo Slurp is the user agent for the web crawler.


10+ Year Member

Msg#: 2476 posted 5:56 am on Jul 7, 2004 (gmt 0)

My site has been in the Yahoo serps, and I type the site:mydomain.com , my homepage is listed there, but it is very old one, almost a few months ago.
Can you tell me why and what happened? How to change this?
Just wait? Or do sth?


10+ Year Member

Msg#: 2476 posted 11:41 am on Jul 7, 2004 (gmt 0)

Tim, your response contradicts Mike's response from few weeks ago in this thread - [webmasterworld.com...] - where he mentioned that both crawlers are feeding the same index. Has this changed in recently?

If FORMERWEBCRAWLER is not feeding the search index, then which index is it feeding?

Thanks for your time.


10+ Year Member

Msg#: 2476 posted 6:49 pm on Jul 9, 2004 (gmt 0)

Can't figure it out.
Submitted new site (optimized, has 27 linked pages to home)to Yahoo (300 bucks), it was included in the directory, but not in search. Google picked it up right away and listed in its search.

It has then still remained far and away by its lonesome in directory (unavailbale even with exact words in search), and suddenly Google has either eliminated it also or bottomed it. I have added many outside links to the site since then as well.

I did this process with a much less important site 4 months ago and it appeared everywhere (Yahoo, Google, searches, directories,etc,) without optimization or links (besides its own 2 pages).

I am marketing with others (adwords, smaller listing directories) but darn it I paid to be included. Similar sites I have seen ranked in the first 2-3 pages are not optimized and NOT listed in the directory. I feel I am being punished for paying for the directory inclusion..

I originally just wanted to be in the Yahoo search results, to heck with the directory.

Thanks for any advice


10+ Year Member

Msg#: 2476 posted 7:56 pm on Jul 9, 2004 (gmt 0)

Submitting to the directory is different than submitting to the search index. We do try to index the directory sites in the web index. To be included in the web search results directly you can use the freeaddurl program, wait until the crawler finds you or use the SiteMatch program.


10+ Year Member

Msg#: 2476 posted 8:24 pm on Jul 12, 2004 (gmt 0)


I have some pages that I've moved over a month ago and they still appear in the directory with high rankings. I have other sites that can't seem to get indexed at all despite high Google SERPs. My stats read that the Inktomi Slurp visits everyday, but it doesn't seem to want to index any subpages pages. What could be the reason for this? How long does it normally take for the Yahoo search to pickup a page?



Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved