homepage Welcome to WebmasterWorld Guest from 54.237.184.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
WordPress
Umbra




msg:4265623
 2:18 pm on Feb 11, 2011 (gmt 0)

How do you handle requests associated with WordPress?

I'm seeing HEAD and GET requests from user agents like

WordPress/3.0.2; http://example.website.com

WordPress/MU; http ://exampleblog.wordpress.com

WordPress.com mShots; http ://support.wordpress.com/contact/
(also mentioned here [webmasterworld.com] )

...Or some standard browser user agent with a referer like http : //exampleblog.wordpress.com

I assume it's a link validation tool, but none of the above confirm exactly what it's looking for, and I'm annoyed that Wordpress doesn't care to provide a link to webmaster documentation.

Some of these Wordpress requests seem to originate from spam-friendly hosts like Planetlab... if a Wordpress hit is blocked with a 403 or 503, and assuming it's not a fake user agent, what happens on the blogger's end?

 

keyplyr




msg:4265807
 7:44 pm on Feb 11, 2011 (gmt 0)

I tried allowing it, assuming it was a link checker. Then I found my content scraped on a wordpress powered blog and dug through my logs to find the scraping event request had WordPress/* as the UA, so I block it now.

incrediBILL




msg:4265819
 8:02 pm on Feb 11, 2011 (gmt 0)

Always blocked.

Umbra




msg:4265839
 8:36 pm on Feb 11, 2011 (gmt 0)

I tried allowing it, assuming it was a link checker. Then I found my content scraped on a wordpress powered blog and dug through my logs to find the scraping event request had WordPress/* as the UA, so I block it now.

I think it can be a link checker, because I looked up the domain in one Wordpress user agent and found an article with a link to our site. On the other hand, there's your example above... If only Wordpress would offer some documentation, I'd know better what to do.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved