Forum Moderators: open

Message Too Old, No Replies

Yahoo crawling a new site

Yahoo is indexing my site but only the home page is getting listed

         

MkSlim

12:55 am on May 8, 2005 (gmt 0)

10+ Year Member



I'm new to this so bear with me.

I can see that Yahoo is visiting my site, but only my home page is getting listed.

How do I recognize that a robot crawled my entire site? I get multiple visits from multiple ip addresses with one hit per visitor, all within a close range. Is that how it should look? Or should it rather be a single visit with multiple hits?

robotsdobetter

4:14 am on May 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Give it a few days and Yahoo should get around to spidering all of your web pages and listing them each in the search results. You can recognize when Yahoo or any robot that spiders your web site by looking at your logs, you can Identify them by looking at their IP addresses or by the user agent name. There is software that will recognize a robot when it visits your web site and will email you.

Getting multiple hits from multiple IP addresses is normal. You will see it sometimes come from just one and others times two. The robots are not connected together, so they don't know they are spidering the same site and the time. To get Yahoo and other search engine robots to visit your web site more you should get more backlinks from web sites that they often visit.

guigy

9:10 pm on May 9, 2005 (gmt 0)



another beginner here folks..

same here, my site has been listed for keywords on the index page for about a month now but the rest of the site isn't spidered by yahoo.

Just a thought but when crosslinking on topic within the site, do hrefs with the full path url give the spider the impression that that it is an external link?

Sorry if this is being stupid but i'm reading and learning very fast...

all help appreciated :)

Thanks
G

jaffstar

9:22 am on May 10, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi Guys,

I recommend you ALL read the following:

[webmasterworld.com...]

yanyading

9:55 am on May 18, 2005 (gmt 0)

10+ Year Member



Yes, my site was crawling (even today) for 3 month eveyday, but only index.asp was listed...