Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Frequency of yahoo crawler

I think we have hit a jackpot :)

1:15 am on Jul 4, 2004 (gmt 0)

10+ Year Member

Yahoo crawler used to come to our site once in few days.. we recently started second round of link development and we got about 20-30 new links in past week.. we had about 50 links earlier.
Since yesterday yahoo crawler is visiting the home page every 3-4 hours.. does that mean we have crossed some link threshold which qualifies us as a better site (compared to before)... we are getting hits from this crawler:

Although this url (HTTP://WWW.ALLTHEWEB.COM/HELP/WEBMASTER/CRAWLER) does not exists, but this is what we are getting in crawler identification.

2:49 am on Jul 7, 2004 (gmt 0)

10+ Year Member

Yes, I think you are so lucky and you did so well! Opposite to your case, Yahoo! crawler haven't come to my site for a long time, even though I update my site frequently.
I don't know how to attract Yahoo! crawler come to get my updated changing? My site get the 3th position on Google, but very low position on Yahoo!
Need your help! Thanks a lot.


5:09 am on Jul 7, 2004 (gmt 0)

10+ Year Member

The "formerwebcrawler" is not the web search crawler. Yahoo Slurp is the user agent for the web crawler.
5:56 am on Jul 7, 2004 (gmt 0)

10+ Year Member

My site has been in the Yahoo serps, and I type the site:mydomain.com , my homepage is listed there, but it is very old one, almost a few months ago.
Can you tell me why and what happened? How to change this?
Just wait? Or do sth?
11:41 am on Jul 7, 2004 (gmt 0)

10+ Year Member

Tim, your response contradicts Mike's response from few weeks ago in this thread - [webmasterworld.com...] - where he mentioned that both crawlers are feeding the same index. Has this changed in recently?

If FORMERWEBCRAWLER is not feeding the search index, then which index is it feeding?

Thanks for your time.

6:49 pm on Jul 9, 2004 (gmt 0)

10+ Year Member

Can't figure it out.
Submitted new site (optimized, has 27 linked pages to home)to Yahoo (300 bucks), it was included in the directory, but not in search. Google picked it up right away and listed in its search.

It has then still remained far and away by its lonesome in directory (unavailbale even with exact words in search), and suddenly Google has either eliminated it also or bottomed it. I have added many outside links to the site since then as well.

I did this process with a much less important site 4 months ago and it appeared everywhere (Yahoo, Google, searches, directories,etc,) without optimization or links (besides its own 2 pages).

I am marketing with others (adwords, smaller listing directories) but darn it I paid to be included. Similar sites I have seen ranked in the first 2-3 pages are not optimized and NOT listed in the directory. I feel I am being punished for paying for the directory inclusion..

I originally just wanted to be in the Yahoo search results, to heck with the directory.

Thanks for any advice


7:56 pm on Jul 9, 2004 (gmt 0)

10+ Year Member

Submitting to the directory is different than submitting to the search index. We do try to index the directory sites in the web index. To be included in the web search results directly you can use the freeaddurl program, wait until the crawler finds you or use the SiteMatch program.
8:24 pm on Jul 12, 2004 (gmt 0)

10+ Year Member


I have some pages that I've moved over a month ago and they still appear in the directory with high rankings. I have other sites that can't seem to get indexed at all despite high Google SERPs. My stats read that the Inktomi Slurp visits everyday, but it doesn't seem to want to index any subpages pages. What could be the reason for this? How long does it normally take for the Yahoo search to pickup a page?




Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month