Welcome to WebmasterWorld Guest from 54.145.95.13

Forum Moderators: martinibuster

Message Too Old, No Replies

Frequency of yahoo crawler

I think we have hit a jackpot :)

     
1:15 am on Jul 4, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 20, 2003
posts:450
votes: 0


Yahoo crawler used to come to our site once in few days.. we recently started second round of link development and we got about 20-30 new links in past week.. we had about 50 links earlier.
Since yesterday yahoo crawler is visiting the home page every 3-4 hours.. does that mean we have crossed some link threshold which qualifies us as a better site (compared to before)... we are getting hits from this crawler:
YAHOO-VERTICALCRAWLER-FORMERWEBCRAWLER/3.9 CRAWLER AT TRD DOT OVERTURE DOT COM; [ALLTHEWEB.COM...]

Although this url (HTTP://WWW.ALLTHEWEB.COM/HELP/WEBMASTER/CRAWLER) does not exists, but this is what we are getting in crawler identification.

2:49 am on July 7, 2004 (gmt 0)

New User

10+ Year Member

joined:June 10, 2004
posts:14
votes: 0


Yes, I think you are so lucky and you did so well! Opposite to your case, Yahoo! crawler haven't come to my site for a long time, even though I update my site frequently.
I don't know how to attract Yahoo! crawler come to get my updated changing? My site get the 3th position on Google, but very low position on Yahoo!
Need your help! Thanks a lot.

Tim

5:09 am on July 7, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 27, 2002
posts:351
votes: 0


The "formerwebcrawler" is not the web search crawler. Yahoo Slurp is the user agent for the web crawler.
5:56 am on July 7, 2004 (gmt 0)

New User

10+ Year Member

joined:June 10, 2004
posts:14
votes: 0


My site has been in the Yahoo serps, and I type the site:mydomain.com , my homepage is listed there, but it is very old one, almost a few months ago.
Can you tell me why and what happened? How to change this?
Just wait? Or do sth?
11:41 am on July 7, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 20, 2003
posts:450
votes: 0


Tim, your response contradicts Mike's response from few weeks ago in this thread - [webmasterworld.com...] - where he mentioned that both crawlers are feeding the same index. Has this changed in recently?

If FORMERWEBCRAWLER is not feeding the search index, then which index is it feeding?

Thanks for your time.
SD

6:49 pm on July 9, 2004 (gmt 0)

New User

10+ Year Member

joined:July 6, 2004
posts:4
votes: 0


Can't figure it out.
Submitted new site (optimized, has 27 linked pages to home)to Yahoo (300 bucks), it was included in the directory, but not in search. Google picked it up right away and listed in its search.

It has then still remained far and away by its lonesome in directory (unavailbale even with exact words in search), and suddenly Google has either eliminated it also or bottomed it. I have added many outside links to the site since then as well.

I did this process with a much less important site 4 months ago and it appeared everywhere (Yahoo, Google, searches, directories,etc,) without optimization or links (besides its own 2 pages).

I am marketing with others (adwords, smaller listing directories) but darn it I paid to be included. Similar sites I have seen ranked in the first 2-3 pages are not optimized and NOT listed in the directory. I feel I am being punished for paying for the directory inclusion..

I originally just wanted to be in the Yahoo search results, to heck with the directory.

Thanks for any advice
TryinTooHard

Tim

7:56 pm on July 9, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 27, 2002
posts:351
votes: 0


Submitting to the directory is different than submitting to the search index. We do try to index the directory sites in the web index. To be included in the web search results directly you can use the freeaddurl program, wait until the crawler finds you or use the SiteMatch program.
Tim
8:24 pm on July 12, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 5, 2004
posts:470
votes: 0


Hi,

I have some pages that I've moved over a month ago and they still appear in the directory with high rankings. I have other sites that can't seem to get indexed at all despite high Google SERPs. My stats read that the Inktomi Slurp visits everyday, but it doesn't seem to want to index any subpages pages. What could be the reason for this? How long does it normally take for the Yahoo search to pickup a page?

Thanks,

Rollo