Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

New Domain - Slurp only visits index and robots.txt



7:15 pm on Nov 8, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Asmentioned in the title, my new domain gets visited by Slurp, but only the index.html and the robots.txt get hit.
I have links to inner pages on the index.html which are followed by other spiders (msnbot, googlebot) and the robots.txt isn't disallowing slurp.
What could be the reason for this "ignorance"?
It seems not to be technical.
any help is veryy much appreciated.


4:12 pm on Nov 9, 2005 (gmt 0)

10+ Year Member

Maybe they think that if Slurp ignores your pages for long enough, you'll sign up for SiteMatch. :)


5:20 pm on Nov 12, 2005 (gmt 0)

10+ Year Member

I had exactly the same problem for a new site that I launched last February. It was fully crawled by Yahoo only after 6 months. Before that, Yahoo crawled only the home page.

It's like the sandbox effect in Google for new sites.


6:16 pm on Nov 12, 2005 (gmt 0)

10+ Year Member

None of my sites (old or new) were DeepCrawled in Oct. Check your stats today, they seem to have started crawling again.

Featured Threads

Hot Threads This Week

Hot Threads This Month