| 2:22 am on Mar 11, 2003 (gmt 0)|
Google Cache's your Robots.txt file, so they only request it once per crawl.
Why not put a robots tag on those pages i.e. <meta name="robots" content="noindex,follow" />
This will stop Google Indexing the Page, but it will still follow the links.
| 2:29 am on Mar 11, 2003 (gmt 0)|
projectphp as you can see the googlebot requested robots.txt just before it crawled the cgi page. The only thing I can think of is that there is more than one instance of a googlebot running per IP, and the request made to the cgi page was from a googlebot instance which requested robots.txt before I updated it.
Good idea to change my cgi pages to like you suggested with the <meta name="robots" content="noindex,follow" />
| 4:46 am on Mar 11, 2003 (gmt 0)|
What are the ip ranges of the visits. I've never seen that "Mediapartners" bit
| 5:11 am on Mar 11, 2003 (gmt 0)|
if you ping crawler9.googlebot.com it is 188.8.131.52
| 5:17 am on Mar 11, 2003 (gmt 0)|
|if you ping crawler9.googlebot.com it is 184.108.40.206 |
I know that, what I'm asking is if the visits with that user agent string are from ip's that are normally associated with googlebot or if someone is masking as Gbot? I'm guessing that 'Mediapartners' is part of a rdns lookup but my stat program doesn't have that functionality to check.
| 5:23 am on Mar 11, 2003 (gmt 0)|
ME TOO! Now I have both meta robots and robots.txt to see if they obey this time.
| 5:54 am on Mar 11, 2003 (gmt 0)|
Powdork, I have no idea. I must admit I thought it was strange with that useragent? Is it possible someone could mask as a googlebot?
| 6:14 am on Mar 11, 2003 (gmt 0)|
kryton, I believe that this user agent is associated with our content ads program. Are you on a site that shows our content ads?
| 3:12 pm on Mar 11, 2003 (gmt 0)|
No GoogleGuy am I not.
| 7:04 pm on Mar 13, 2003 (gmt 0)|
I'm seeing this mediapartners bot, too.
See my post here:
I am not doing anything with "content ads" that I know of (unless my ad network is serving them up somehow)?
GoogleGuy, what's this about and why is this bot so aggressive?
| 7:10 pm on Mar 13, 2003 (gmt 0)|
Us too - we've got the Google MediaPartners bot and have been wondering what it is... We are not involved with the google ads in any way (at this point, only overture).
| 8:40 pm on Mar 13, 2003 (gmt 0)|
kryton, do you sell any banner ads? Content ads can also show up on your site that way. I believe the Mediapartners user agent is the bot for content ads, but it would be helpful if anyone could mention specific urls. I guess that's out of bounds, but I'm pretty sure that these are for content ads.
| 9:12 pm on Mar 13, 2003 (gmt 0)|
Wouldn't the ip ranges of the bots give all the needed info?
| 11:27 pm on Mar 13, 2003 (gmt 0)|
We use a little cgi script to rotate through some 234x60 banners - but it's only informational internal ads that link to different pages on our site - nothing external.
I've heard your stickymail is turned off - how should I send you our URL?
| 2:25 am on Mar 14, 2003 (gmt 0)|
Mmm. Do it as a spam report and mention your nick and this url. I'll get it. Thanks..
| 3:12 am on Mar 14, 2003 (gmt 0)|
Do people really send in their own URLs to the spam report when you ask them? I try to keep everything above board but that would still make me a tad nervous.;)
| 3:53 am on Mar 14, 2003 (gmt 0)|
I'm seeing the bot also although it is behaving itself. And though I don't serve Google content ads (yet), I'm certainly interested.