Welcome to WebmasterWorld Guest from 54.145.44.134

Forum Moderators: mademetop

Message Too Old, No Replies

Brand new website, popular topic

     

unlimited

8:42 am on Aug 29, 2005 (gmt 0)

5+ Year Member



My brank new website, about jokes, funny pictures, games, etc. got online 2 days ago. I checked the log file, find google, yahoo, and msn bot came. google and yahoo bots crawled thousands pages, but why MSN bot only crawled robots.txt, index.html and rss.xml?

robotsdobetter

6:33 pm on Aug 29, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



There's no telling why bots do the things they do sometimes. It's not like they have a mind like ours (For now anyway), that can learn as it goes. Sometimes they will spider 100s of web pages and other days only a few. Give it some time and it will get around to your other web pages. Just be happy it knows your site is there.

Gargen

3:47 am on Aug 30, 2005 (gmt 0)

10+ Year Member



Once google just wouldnt leave site alone 50k pages in a single day that means it was spidering multiple times, it was madness msn has done a similar thing but not on the same level, you will see very soon SE like G and msn are very unpredictible
 

Featured Threads

Hot Threads This Week

Hot Threads This Month