homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

New Domain - Slurp only visits index and robots.txt

 7:15 pm on Nov 8, 2005 (gmt 0)

Asmentioned in the title, my new domain gets visited by Slurp, but only the index.html and the robots.txt get hit.
I have links to inner pages on the index.html which are followed by other spiders (msnbot, googlebot) and the robots.txt isn't disallowing slurp.
What could be the reason for this "ignorance"?
It seems not to be technical.
any help is veryy much appreciated.



 4:12 pm on Nov 9, 2005 (gmt 0)

Maybe they think that if Slurp ignores your pages for long enough, you'll sign up for SiteMatch. :)


 5:20 pm on Nov 12, 2005 (gmt 0)

I had exactly the same problem for a new site that I launched last February. It was fully crawled by Yahoo only after 6 months. Before that, Yahoo crawled only the home page.

It's like the sandbox effect in Google for new sites.


 6:16 pm on Nov 12, 2005 (gmt 0)

None of my sites (old or new) were DeepCrawled in Oct. Check your stats today, they seem to have started crawling again.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved