homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

When will Yahoo spider all my site?
8 months.

 8:53 am on May 8, 2004 (gmt 0)

Yahoo has only indexed my homepage, which was created in October 2003. I have approx 70 pages of reasonable content and about 50 incoming links - all to my homepage.
When will Yahoo index my other pages, or do I need to submit them all?
Any guidance is appreciated.



 10:55 pm on May 10, 2004 (gmt 0)

What kind of inbound links do you have for your site? If there are no problems with your site, and you have good inbound links, Slurp should be spidering you frequently.


 11:34 pm on May 10, 2004 (gmt 0)

I have almost the same problem. Yahoo reports 154 links to my site, of which about 125 are from duplicates of Zeal and ODP. The rest of my links are legitimate links from other sites, some reciprocal, some not. No link farms.

Yahoo has included 5 of my pages so far. It has been adding them one at a time every couple of weeks. Is there anything other than pay-for-inclusion I can do to make Yahoo spider more of my pages?




 11:50 pm on May 10, 2004 (gmt 0)

What seems to work for me is to get external inbound links pointing to my interior pages.

Also take a look at the link structure within your site to make sure that the pages you want spidered are not to far off the root.


 12:10 am on May 11, 2004 (gmt 0)

Thanks for the advice pmac, I'm finding slurp isn't too thirsty these days either...


 8:33 am on May 11, 2004 (gmt 0)

I had been searching for my Yahoo indexed pages with
which only showed my homepage

I now realise that the correct sytax should be


This shows 50+ of my pages and so I am already indexed.


 12:13 pm on May 11, 2004 (gmt 0)

I've had almost 3000 pages on my site since January and Yahoo still indexes just 139 of them. Hundreds of incoming links, Google PR7. Slurp just doesn't do crawling very well.


 12:50 pm on May 11, 2004 (gmt 0)

i have the same problem 200 000+ pages and slurp still get my robots.txt and index.html page 20 times/day and never goo deeply.. all the pages are in the root directory and are linked correctly.. i haven't hidden text or link, no popup..

i'm going crazy


 2:43 pm on May 11, 2004 (gmt 0)

you have 200,000 files just in 1 directory?

I guess you are the same guy I heard about before that had 20,000 emails in his inbox.


 3:09 pm on May 11, 2004 (gmt 0)

no i'm not the same guy, but yes i have 200 000 files in root directory.. i have my own server and i'm programmer i can do this and more :)


 9:46 pm on May 11, 2004 (gmt 0)

Can anybody tell me what this means? Every other day Yahoo! Slurp visits my site and this is what shows up in the logs.

2004-05-05 10:36:14 - W3SVC33 VEGA 80 GET /robots.txt - 200 0 421 197 31 HTTP/1.0 www.widgets.com
Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp) - -
2004-05-05 10:36:14 - W3SVC33 VEGA 80 GET /index.html - 200 0 542 187 32 HTTP/1.0 www.widgets.com
Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp) - -


 4:52 am on May 17, 2004 (gmt 0)

I'm having a similar problem getting all my pages in the index. For about two months now Yahoo seems to have all my pages in, but only displays about 3/4 of them. Whenever I hit "more pages from this site" it says the correct number are there, but looking at the results only yields 3/4 or less. What gives? When will all the pages show up?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved