homepage Welcome to WebmasterWorld Guest from 54.211.73.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
Controlling Bingbot crawling
bingdude



 
Msg#: 4335903 posted 5:51 pm on Jul 6, 2011 (gmt 0)

Just a quick note to remind folks that YOU can control the crawl rate of Bingbot through Bing's Webmaster Tools.

You can tell Bingbot to crawler slower or faster, and set those rates per hour, over the entire day. So, with these controls, you can tailor when Bingbot crawls you and at what pace.

This is great for managing crawling v. business loads. Tell Bingbot to crawl you slowly during peak business times, and ramp up crawling when you have more bandwidth available.

We do honor the crawl pofiles you set via this tool.

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4335903 posted 7:58 pm on Jul 6, 2011 (gmt 0)

Hi, Bingdude.

Could I ask you to look at a thread re: crawl rate in the "Search Engine Spider and User Agent Identification" forum herabouts, please?

The topic is "MSN's many cloaked bots. Again". It addresses several bingbot problems including invalid rDNS and UAs and (later in the thread) crawl rate.

[webmasterworld.com...]

bingdude



 
Msg#: 4335903 posted 8:17 pm on Jul 6, 2011 (gmt 0)

That thread has been pointed out to me. It's also from last August, mainly, from what I was seeing. This morning I was only seeing three fresh posts (from this month) and only one of which mentioned Bingbot or MSNbot.

Spoke too soon - I see my error now - still getting used to the functionality of the board here. ;) I'm seeing the recent, other posts now... ;)

duane

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4335903 posted 10:10 pm on Jul 6, 2011 (gmt 0)

Takes a while, don't it? :)

My own problem, noted in that thread, is the number of visits I get from otherwise legitmate-seeming bingbots but with incorrect rDNS. The promise made on your bots page is (was?) that the rDNS would resolve to a reasonable URL that identified the IP as a bot.

Have to say you're not alone in this but I am rejecting a LOT of bot hits through this at a time when I'm trying to be nice to Bing. :)

Apart from that, it's nice to see someone from Bing taking an interest in our discussions. I hope this is not ephemeral, which it seems to be with another well-known SE. :)

scooterdude



 
Msg#: 4335903 posted 11:31 pm on Jul 6, 2011 (gmt 0)

perhaps you could shed some light on what attracts a deeper crawl and more indexing from bingbot :)

bingdude



 
Msg#: 4335903 posted 12:18 am on Jul 7, 2011 (gmt 0)

@dstiles

Yeah, but I did used to hang about here quite a bunch personally years ago, so I should have been able to recall the situation. ;)

Now, any chance you can point me to a URL with that statement around rDNS lookups resolving in a certain manner? If I have something solid I can grab my toga and run through the halls here with a lit torch looking for an owner to discuss the situation with. :)

As for being around persistantly, you'll find sometimes I disappear, but that's mostly because I travel a bunch for Bing. To be honest, I love hanging in the forums with my peeps, as this is where I came from (I used to mod over at Search Engine Forums back in the day). :)

Couple of ways to reach me if you don't find me responsive here:

1 - post up in the Bing Webmaster forums - that I partol as a function of my daily work. Plus we have support staff there who can alert me to items people post specific to me, when they don't see an answer quickly.

2 - @duaneforrester - now, you may not like the short responses, because with only so many characters there's often not enough room to find a polite way to say "your baby is ugly", if such should be the case. ;)

As an FYI, I was an inhouse SEO for over a decade before taking this role with Bing Webmaster, so the pains you guys feel, I understand first hand.

@scooterdude

Yep...

1 - quality, unique content
2 - fresh, quality, unique content
3 - sites that users frequent tend to fair better
4 - skip the tricks - we've seen them all, and while 99% of them we simply let slide, you'd hate to hit that 1% category
5 - social signals are starting to matter more - can't really say how much more, but look at it this way - if you're active socially, and users like you and visit your site, we see their actions
6 - look into rich snippets [schema.org]
7 - yes, inbound links matter - focus on quality instead of quantity. How few links does it take to help rank a page of yours well? Often very few. ;)
8 - Bing doc on successful indexing [onlinehelp.microsoft.com] with us
9 - Bing doc on inserting individual URLs [onlinehelp.microsoft.com] into our index
10 - whole mess of other docs [onlinehelp.microsoft.com] related to using the Bing Webmaster tools, indexing, seo and more

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4335903 posted 10:35 pm on Jul 7, 2011 (gmt 0)

Bingdude - thanks for the response.

Bad rDNS - 157.55.16.0 - 157.55.18.255 is a good place to begin - there may be others but those are the biggest hitters. Some of those IPs have been actively bot'ing since October 2010 right up to date. As far as I have checked, none in that range carry the bot rDNS: they are generally (all?) msft.msnhst.etc. I did a DNS scan and grepped about a month ago but found no crawl references.

bingdude



 
Msg#: 4335903 posted 10:50 pm on Jul 7, 2011 (gmt 0)

Checking with the Bingbot wranglers now - not sure when I'll get a response, but I'll post up as soon as I can.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4335903 posted 9:56 pm on Jul 8, 2011 (gmt 0)

Thanks. Look forward to the results. :)

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4335903 posted 4:46 pm on Jul 9, 2011 (gmt 0)

Just remembered the other bot complaint, bingdude - badly formed UA coming from a legit IP...

msnbot/2.0b (+http://search.msn.com/msnbot.htm)._

The trailing dot/Underscore causes the access to be rejected.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4335903 posted 8:21 pm on Jul 9, 2011 (gmt 0)

If you can remember any of the relevant IPs or rough IP ranges, that might help a bit more.

Perhaps we can collect all further issues in this other thread: [webmasterworld.com...]

It looks like one reported issue may already be on the way to being fixed.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4335903 posted 9:15 pm on Jul 9, 2011 (gmt 0)

The incorrect UAs seem to occur on several bot IPs but today they came in on the range 157.55.116.7 - 157.55.116.97 (specifically, 157.55.116.23 but maybe others)

I've reported most of this in the other forum already.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4335903 posted 9:58 pm on Aug 4, 2011 (gmt 0)

Bingdude - any response yet on the bot IPs and UAs?

bingdude



 
Msg#: 4335903 posted 1:01 am on Aug 5, 2011 (gmt 0)

Nothing I can report out on yet, dstiles. Not a dead issue on my end, just in the (large)stack for review. ;)

duane

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4335903 posted 9:42 pm on Aug 5, 2011 (gmt 0)

Thanks. A stick of dynamite any use? :)

bingdude



 
Msg#: 4335903 posted 5:29 pm on Aug 9, 2011 (gmt 0)

Maybe for clearing off my desk... ;)

tangor

WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4335903 posted 7:07 pm on Aug 9, 2011 (gmt 0)

A breath of fresh air has entered the discussions! As some of these issues have been around nearly a year, any change soon(est) would be greatly appreciated!

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4335903 posted 7:24 pm on Aug 9, 2011 (gmt 0)

Bing! Bang! Boom!

bingdude



 
Msg#: 4335903 posted 11:07 pm on Aug 9, 2011 (gmt 0)

HOLY CRAP! There's a phone on my desk?!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved