homepage Welcome to WebmasterWorld Guest from 54.166.173.147
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
Controlling Bingbot crawling
bingdude




msg:4335905
 5:51 pm on Jul 6, 2011 (gmt 0)

Just a quick note to remind folks that YOU can control the crawl rate of Bingbot through Bing's Webmaster Tools.

You can tell Bingbot to crawler slower or faster, and set those rates per hour, over the entire day. So, with these controls, you can tailor when Bingbot crawls you and at what pace.

This is great for managing crawling v. business loads. Tell Bingbot to crawl you slowly during peak business times, and ramp up crawling when you have more bandwidth available.

We do honor the crawl pofiles you set via this tool.

 

dstiles




msg:4335979
 7:58 pm on Jul 6, 2011 (gmt 0)

Hi, Bingdude.

Could I ask you to look at a thread re: crawl rate in the "Search Engine Spider and User Agent Identification" forum herabouts, please?

The topic is "MSN's many cloaked bots. Again". It addresses several bingbot problems including invalid rDNS and UAs and (later in the thread) crawl rate.

[webmasterworld.com...]

bingdude




msg:4335991
 8:17 pm on Jul 6, 2011 (gmt 0)

That thread has been pointed out to me. It's also from last August, mainly, from what I was seeing. This morning I was only seeing three fresh posts (from this month) and only one of which mentioned Bingbot or MSNbot.

Spoke too soon - I see my error now - still getting used to the functionality of the board here. ;) I'm seeing the recent, other posts now... ;)

duane

dstiles




msg:4336049
 10:10 pm on Jul 6, 2011 (gmt 0)

Takes a while, don't it? :)

My own problem, noted in that thread, is the number of visits I get from otherwise legitmate-seeming bingbots but with incorrect rDNS. The promise made on your bots page is (was?) that the rDNS would resolve to a reasonable URL that identified the IP as a bot.

Have to say you're not alone in this but I am rejecting a LOT of bot hits through this at a time when I'm trying to be nice to Bing. :)

Apart from that, it's nice to see someone from Bing taking an interest in our discussions. I hope this is not ephemeral, which it seems to be with another well-known SE. :)

scooterdude




msg:4336086
 11:31 pm on Jul 6, 2011 (gmt 0)

perhaps you could shed some light on what attracts a deeper crawl and more indexing from bingbot :)

bingdude




msg:4336098
 12:18 am on Jul 7, 2011 (gmt 0)

@dstiles

Yeah, but I did used to hang about here quite a bunch personally years ago, so I should have been able to recall the situation. ;)

Now, any chance you can point me to a URL with that statement around rDNS lookups resolving in a certain manner? If I have something solid I can grab my toga and run through the halls here with a lit torch looking for an owner to discuss the situation with. :)

As for being around persistantly, you'll find sometimes I disappear, but that's mostly because I travel a bunch for Bing. To be honest, I love hanging in the forums with my peeps, as this is where I came from (I used to mod over at Search Engine Forums back in the day). :)

Couple of ways to reach me if you don't find me responsive here:

1 - post up in the Bing Webmaster forums - that I partol as a function of my daily work. Plus we have support staff there who can alert me to items people post specific to me, when they don't see an answer quickly.

2 - @duaneforrester - now, you may not like the short responses, because with only so many characters there's often not enough room to find a polite way to say "your baby is ugly", if such should be the case. ;)

As an FYI, I was an inhouse SEO for over a decade before taking this role with Bing Webmaster, so the pains you guys feel, I understand first hand.

@scooterdude

Yep...

1 - quality, unique content
2 - fresh, quality, unique content
3 - sites that users frequent tend to fair better
4 - skip the tricks - we've seen them all, and while 99% of them we simply let slide, you'd hate to hit that 1% category
5 - social signals are starting to matter more - can't really say how much more, but look at it this way - if you're active socially, and users like you and visit your site, we see their actions
6 - look into rich snippets [schema.org]
7 - yes, inbound links matter - focus on quality instead of quantity. How few links does it take to help rank a page of yours well? Often very few. ;)
8 - Bing doc on successful indexing [onlinehelp.microsoft.com] with us
9 - Bing doc on inserting individual URLs [onlinehelp.microsoft.com] into our index
10 - whole mess of other docs [onlinehelp.microsoft.com] related to using the Bing Webmaster tools, indexing, seo and more

dstiles




msg:4336667
 10:35 pm on Jul 7, 2011 (gmt 0)

Bingdude - thanks for the response.

Bad rDNS - 157.55.16.0 - 157.55.18.255 is a good place to begin - there may be others but those are the biggest hitters. Some of those IPs have been actively bot'ing since October 2010 right up to date. As far as I have checked, none in that range carry the bot rDNS: they are generally (all?) msft.msnhst.etc. I did a DNS scan and grepped about a month ago but found no crawl references.

bingdude




msg:4336674
 10:50 pm on Jul 7, 2011 (gmt 0)

Checking with the Bingbot wranglers now - not sure when I'll get a response, but I'll post up as soon as I can.

dstiles




msg:4337174
 9:56 pm on Jul 8, 2011 (gmt 0)

Thanks. Look forward to the results. :)

dstiles




msg:4337424
 4:46 pm on Jul 9, 2011 (gmt 0)

Just remembered the other bot complaint, bingdude - badly formed UA coming from a legit IP...

msnbot/2.0b (+http://search.msn.com/msnbot.htm)._

The trailing dot/Underscore causes the access to be rejected.

g1smd




msg:4337478
 8:21 pm on Jul 9, 2011 (gmt 0)

If you can remember any of the relevant IPs or rough IP ranges, that might help a bit more.

Perhaps we can collect all further issues in this other thread: [webmasterworld.com...]

It looks like one reported issue may already be on the way to being fixed.

dstiles




msg:4337510
 9:15 pm on Jul 9, 2011 (gmt 0)

The incorrect UAs seem to occur on several bot IPs but today they came in on the range 157.55.116.7 - 157.55.116.97 (specifically, 157.55.116.23 but maybe others)

I've reported most of this in the other forum already.

dstiles




msg:4347970
 9:58 pm on Aug 4, 2011 (gmt 0)

Bingdude - any response yet on the bot IPs and UAs?

bingdude




msg:4348043
 1:01 am on Aug 5, 2011 (gmt 0)

Nothing I can report out on yet, dstiles. Not a dead issue on my end, just in the (large)stack for review. ;)

duane

dstiles




msg:4348420
 9:42 pm on Aug 5, 2011 (gmt 0)

Thanks. A stick of dynamite any use? :)

bingdude




msg:4349588
 5:29 pm on Aug 9, 2011 (gmt 0)

Maybe for clearing off my desk... ;)

tangor




msg:4349637
 7:07 pm on Aug 9, 2011 (gmt 0)

A breath of fresh air has entered the discussions! As some of these issues have been around nearly a year, any change soon(est) would be greatly appreciated!

g1smd




msg:4349640
 7:24 pm on Aug 9, 2011 (gmt 0)

Bing! Bang! Boom!

bingdude




msg:4349699
 11:07 pm on Aug 9, 2011 (gmt 0)

HOLY CRAP! There's a phone on my desk?!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved