Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

New Domain - Slurp only visits index and robots.txt

7:15 pm on Nov 8, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 27, 2002
votes: 0

Asmentioned in the title, my new domain gets visited by Slurp, but only the index.html and the robots.txt get hit.
I have links to inner pages on the index.html which are followed by other spiders (msnbot, googlebot) and the robots.txt isn't disallowing slurp.
What could be the reason for this "ignorance"?
It seems not to be technical.
any help is veryy much appreciated.
4:12 pm on Nov 9, 2005 (gmt 0)

New User

10+ Year Member

joined:Nov 25, 2003
votes: 0

Maybe they think that if Slurp ignores your pages for long enough, you'll sign up for SiteMatch. :)
5:20 pm on Nov 12, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Jan 27, 2004
votes: 0

I had exactly the same problem for a new site that I launched last February. It was fully crawled by Yahoo only after 6 months. Before that, Yahoo crawled only the home page.

It's like the sandbox effect in Google for new sites.

6:16 pm on Nov 12, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Apr 17, 2002
votes: 0

None of my sites (old or new) were DeepCrawled in Oct. Check your stats today, they seem to have started crawling again.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members