Forum Moderators: goodroi

Message Too Old, No Replies

Geting lots of hits on robots.txt but not much indexes in Yahoo/Google

         

ball4121

1:55 am on Nov 25, 2005 (gmt 0)

10+ Year Member



Ya i'm getting helacious (sp?) hits on my robots.txt file by both SE but not getting indexed, i'm getting so many indexes on MSN.

Googlebot ¦ 366+7 ¦ 24 Nov 2005 - 15:58
Inktomi Slurp ¦ 31+32 ¦ 24 Nov 2005 - 03:47

Number after plus is hits on robots.txt

What's the deal?

Marcia

2:04 am on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



There's a pattern to how Google hits robots.txt and it isn't every time. That figure doesn't look bad, considering. With Slurp, that "equal" figure is normal, afaik, when they're not crawling a site past the homepage.

How much of the site shows up in the index with a site:example.com search?

ball4121

3:13 am on Nov 25, 2005 (gmt 0)

10+ Year Member



A google site:domain.com I get 1 result.

A yahoo site:domain.com I get 2 results. (old index that doesn't exist, and my current forums.

A MSN site:domain.com I get 45 results. LOL.

The site was published on the web late late September/ early early October.

Marcia

3:27 am on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Check to see that your robots.txt validates and see if you can find any prior history on the domain at the Wayback Machine.

You can also check to see if there are any tacky sites on the same IP number, it's not impossible that it's a server-related issue. Not only a "tainted IP" but sometimes hosts do things to turn bots away. When you find other sites/domains on the IP, check to see if they're fully indexed. You can also ask the host.

Stefan

3:28 am on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The site was published on the web late late September/ early early October.

It's a bit early to see many listings. MSN is the freshest for new sites these days, Y has always been rather slow, and G has a problem with listing new sites (old sites, new pages, no problem). Keep adding content in the interim, and you should see improvement eventually (they're checking the robots.txt, so they know you're alive).

ball4121

7:21 am on Nov 25, 2005 (gmt 0)

10+ Year Member



Yes Marcia my robots.txt validates with no errors

and Stefan thank you for the advice i'll just kick back and ride the wave.

Marcia

8:04 am on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



And get some quality inbound links, while you're at it.

ball4121

4:47 am on Nov 26, 2005 (gmt 0)

10+ Year Member



WOA! Google just indexed 3 pages worth of my site.