Welcome to WebmasterWorld Guest from 54.162.240.235

Forum Moderators: mack

Message Too Old, No Replies

Controlling Bingbot crawling

     
5:51 pm on Jul 6, 2011 (gmt 0)



Just a quick note to remind folks that YOU can control the crawl rate of Bingbot through Bing's Webmaster Tools.

You can tell Bingbot to crawler slower or faster, and set those rates per hour, over the entire day. So, with these controls, you can tailor when Bingbot crawls you and at what pace.

This is great for managing crawling v. business loads. Tell Bingbot to crawl you slowly during peak business times, and ramp up crawling when you have more bandwidth available.

We do honor the crawl pofiles you set via this tool.
7:58 pm on Jul 6, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Hi, Bingdude.

Could I ask you to look at a thread re: crawl rate in the "Search Engine Spider and User Agent Identification" forum herabouts, please?

The topic is "MSN's many cloaked bots. Again". It addresses several bingbot problems including invalid rDNS and UAs and (later in the thread) crawl rate.

[webmasterworld.com...]
8:17 pm on Jul 6, 2011 (gmt 0)



That thread has been pointed out to me. It's also from last August, mainly, from what I was seeing. This morning I was only seeing three fresh posts (from this month) and only one of which mentioned Bingbot or MSNbot.

Spoke too soon - I see my error now - still getting used to the functionality of the board here. ;) I'm seeing the recent, other posts now... ;)

duane
10:10 pm on Jul 6, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Takes a while, don't it? :)

My own problem, noted in that thread, is the number of visits I get from otherwise legitmate-seeming bingbots but with incorrect rDNS. The promise made on your bots page is (was?) that the rDNS would resolve to a reasonable URL that identified the IP as a bot.

Have to say you're not alone in this but I am rejecting a LOT of bot hits through this at a time when I'm trying to be nice to Bing. :)

Apart from that, it's nice to see someone from Bing taking an interest in our discussions. I hope this is not ephemeral, which it seems to be with another well-known SE. :)
11:31 pm on Jul 6, 2011 (gmt 0)



perhaps you could shed some light on what attracts a deeper crawl and more indexing from bingbot :)
12:18 am on Jul 7, 2011 (gmt 0)



@dstiles

Yeah, but I did used to hang about here quite a bunch personally years ago, so I should have been able to recall the situation. ;)

Now, any chance you can point me to a URL with that statement around rDNS lookups resolving in a certain manner? If I have something solid I can grab my toga and run through the halls here with a lit torch looking for an owner to discuss the situation with. :)

As for being around persistantly, you'll find sometimes I disappear, but that's mostly because I travel a bunch for Bing. To be honest, I love hanging in the forums with my peeps, as this is where I came from (I used to mod over at Search Engine Forums back in the day). :)

Couple of ways to reach me if you don't find me responsive here:

1 - post up in the Bing Webmaster forums - that I partol as a function of my daily work. Plus we have support staff there who can alert me to items people post specific to me, when they don't see an answer quickly.

2 - @duaneforrester - now, you may not like the short responses, because with only so many characters there's often not enough room to find a polite way to say "your baby is ugly", if such should be the case. ;)

As an FYI, I was an inhouse SEO for over a decade before taking this role with Bing Webmaster, so the pains you guys feel, I understand first hand.

@scooterdude

Yep...

1 - quality, unique content
2 - fresh, quality, unique content
3 - sites that users frequent tend to fair better
4 - skip the tricks - we've seen them all, and while 99% of them we simply let slide, you'd hate to hit that 1% category
5 - social signals are starting to matter more - can't really say how much more, but look at it this way - if you're active socially, and users like you and visit your site, we see their actions
6 - look into rich snippets [schema.org]
7 - yes, inbound links matter - focus on quality instead of quantity. How few links does it take to help rank a page of yours well? Often very few. ;)
8 - Bing doc on successful indexing [onlinehelp.microsoft.com] with us
9 - Bing doc on inserting individual URLs [onlinehelp.microsoft.com] into our index
10 - whole mess of other docs [onlinehelp.microsoft.com] related to using the Bing Webmaster tools, indexing, seo and more
10:35 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Bingdude - thanks for the response.

Bad rDNS - 157.55.16.0 - 157.55.18.255 is a good place to begin - there may be others but those are the biggest hitters. Some of those IPs have been actively bot'ing since October 2010 right up to date. As far as I have checked, none in that range carry the bot rDNS: they are generally (all?) msft.msnhst.etc. I did a DNS scan and grepped about a month ago but found no crawl references.
10:50 pm on Jul 7, 2011 (gmt 0)



Checking with the Bingbot wranglers now - not sure when I'll get a response, but I'll post up as soon as I can.
9:56 pm on Jul 8, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Thanks. Look forward to the results. :)
4:46 pm on Jul 9, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Just remembered the other bot complaint, bingdude - badly formed UA coming from a legit IP...

msnbot/2.0b (+http://search.msn.com/msnbot.htm)._

The trailing dot/Underscore causes the access to be rejected.
8:21 pm on Jul 9, 2011 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



If you can remember any of the relevant IPs or rough IP ranges, that might help a bit more.

Perhaps we can collect all further issues in this other thread: [webmasterworld.com...]

It looks like one reported issue may already be on the way to being fixed.
9:15 pm on Jul 9, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



The incorrect UAs seem to occur on several bot IPs but today they came in on the range 157.55.116.7 - 157.55.116.97 (specifically, 157.55.116.23 but maybe others)

I've reported most of this in the other forum already.
9:58 pm on Aug 4, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Bingdude - any response yet on the bot IPs and UAs?
1:01 am on Aug 5, 2011 (gmt 0)



Nothing I can report out on yet, dstiles. Not a dead issue on my end, just in the (large)stack for review. ;)

duane
9:42 pm on Aug 5, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Thanks. A stick of dynamite any use? :)
5:29 pm on Aug 9, 2011 (gmt 0)



Maybe for clearing off my desk... ;)
7:07 pm on Aug 9, 2011 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



A breath of fresh air has entered the discussions! As some of these issues have been around nearly a year, any change soon(est) would be greatly appreciated!
7:24 pm on Aug 9, 2011 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Bing! Bang! Boom!
11:07 pm on Aug 9, 2011 (gmt 0)



HOLY CRAP! There's a phone on my desk?!
 

Featured Threads

Hot Threads This Week

Hot Threads This Month