Slurp doesn't like subpages

Or just a matter of time?

[ThomasB]

I have a site with 5 subpages, which are all interlinked. The site has plenty of links from different sources.

All 5 sites where listed on Google since about 3 weeks, but on Y I just saw the homepage. After checking the logs I realized that slurp is coming at least once a day, but never spidered any subpages. A look into the sourcecode showed me, that I forgot the '"' (only <a href=page.html>Page</a>) so I added them 2 days ago. Slurp came 3 times since I made the change, but I still can't see it on any subpage.

Any suggestions?

[2_much]

Hi ThomasB, I'm not sure I'm following your question. Can you re-phrase it?

[webzilla]

Apparently he is saying that Yahoo is not spidering any deeper than his home page.

If it behaves anything like Inktomi did, there is no way of knowing when or if it will do a deep crawl. On some sites it will get right to it within a week or two, other sites several months before a deep crawl (most typical) and on some it may never go any deeper than the home page.

[ThomasB]

sorry if my problem was not understandable. Once again:

I have a 4 weeks old site with 5 subpages. The homepage gets spidered once or twice a day, but the subpages don't get any visits from Slurp.

I discovered that I had a problem in the links because they looked like <a href=subpage.html> instead of <a href="subpage.html"> I changed that about 1 week ago but slurp still spiders the homepage only but doesn't look at the subpages.

[2_much]

Ok I get it. As the site is new I wouldn't think it has a penalty. It could just be that it'll take Slurp some time to come back for the deeper content.

If you're not paying for SiteMatch, you need to make sure you get links from sites in Yahoo's "chore". Once that happens, if you're not paying, it still takes longer for them to spider you - between 10 - 14 days.

[PatrickDeese]

From what I've seen, it seems that Y! likes to do deep spidering in bursts, like the old Google deep crawl.

[ThomasB]

Patrick, if that's the case I might wait a couple of days to see what happens. I hope Slurp will visit them soon.

[GeordieSEO]

Slurp has always been notoriously slow for doing deep crawls. I think the newer one isn't supposed to be as bad.

[atlrus]

Well, it doesnt matter if slurp will crawl your pages or not. I have over 2,000 pages crawled in my log by slurp, but yahoo results still show only 5 pages of my website.

[walmslei]

Further to this discussion, we had a site that Yahoo visited for over a month just fetching the home page (index.htm) and also Robots.txt

I emailed the [e-mail] address to see if my site was penalised or if there was anything amiss.

Yahoo replied within a week, stating they checked the structure of our site, and it seemed to be OK and if I was patient the spider would eventually visit.

The following day we were spidered in full, and indexed within a couple of days for over 90% of the pages.

I do not know if the email helped.....