homepage Welcome to WebmasterWorld Guest from 50.17.27.205
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

    
New Domain - Slurp only visits index and robots.txt
onlineleben




msg:822364
 7:15 pm on Nov 8, 2005 (gmt 0)

Asmentioned in the title, my new domain gets visited by Slurp, but only the index.html and the robots.txt get hit.
I have links to inner pages on the index.html which are followed by other spiders (msnbot, googlebot) and the robots.txt isn't disallowing slurp.
What could be the reason for this "ignorance"?
It seems not to be technical.
any help is veryy much appreciated.
Thanks

 

hmatisse




msg:822365
 4:12 pm on Nov 9, 2005 (gmt 0)

Maybe they think that if Slurp ignores your pages for long enough, you'll sign up for SiteMatch. :)

aris1970




msg:822366
 5:20 pm on Nov 12, 2005 (gmt 0)

I had exactly the same problem for a new site that I launched last February. It was fully crawled by Yahoo only after 6 months. Before that, Yahoo crawled only the home page.

It's like the sandbox effect in Google for new sites.

Tigrou




msg:822367
 6:16 pm on Nov 12, 2005 (gmt 0)

None of my sites (old or new) were DeepCrawled in Oct. Check your stats today, they seem to have started crawling again.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved