Welcome to WebmasterWorld Guest from 54.159.44.227

Forum Moderators: goodroi

Message Too Old, No Replies

WP blog traffic plummeting. Bad robots.txt?

     

Boulder90

7:12 pm on May 2, 2009 (gmt 0)

5+ Year Member



Thanks for any help on this one.

My hobby blog gets about 1k visitors a day and on April 18th it started plummeting down to 300 a day. I have recently implemented this robots.txt file for controlling duplicate content on this wordpress blog. Am I somehow blocking other sites abilities to grab my feeds? Thanks

Actual file:

User-agent: *
Disallow:

Disallow: /wp-
Disallow: /search
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$

[edited by: Boulder90 at 7:13 pm (utc) on May 2, 2009]

g1smd

7:47 pm on May 2, 2009 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I would use only ONE single * in any one rule.

I'd imagine a parser choking on /*/*/*/

With simplification you only need six rules (or so).

jbinbpt

7:57 pm on May 2, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Did you run it through Google Webmaster tools Analyze Robots.txt?

Boulder90

9:56 pm on May 2, 2009 (gmt 0)

5+ Year Member



Thanks g1smd. I'm getting the feeling that I am blocking feeds with my file which is why the traffic is way down. Are feeds really considered dupe content by Google?

jbin-

Yeah Google says it is fine.

Boulder90

5:06 am on May 3, 2009 (gmt 0)

5+ Year Member



G1smd -

How does this look?

Disallow: /wp-
Disallow: /search
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$

g1smd

5:46 pm on May 3, 2009 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



You can delete the exact repetition of rules. You'll be left with about eight.

Boulder90

7:08 pm on May 8, 2009 (gmt 0)

5+ Year Member



My traffic keeps plummeting, and google has droppd 70 pages from the non-supplemental and almost all of my images from the blog. Today I noticed that most of my traffic the last few days has come from some sort of russian blog with an emphasis on sex links. However, I could NOT find my link on that page even though Awstats say that's exactly where it's coming from.

What the heck is going on? There's no malicious code on my p[agesand I am certainly not placing my link on russion blogs with links to all things sex.

londrum

7:58 pm on May 8, 2009 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



it might be one of those proxy hijacking things
try doing a search in google for inurl:example.com
where example.com is your domain name.

you want to look for loads of foreign pages where your url is part of the query string.

there's a long post on this site somewhere about how to stop it. can't remember exactly where, but a search for proxy hijacking should find it (if that's what it turns out to be)

Boulder90

2:13 am on May 9, 2009 (gmt 0)

5+ Year Member



Thanks for the comments, londrum. I checked that test and no luck. I have no idea what's going on. I've dropped 70% of my traffic, Google dropped my images and 70 pages of non-supps. The only thing I can point to is that the page responsible for most of my traffic in May is bizarre.

Boulder90

3:56 am on May 10, 2009 (gmt 0)

5+ Year Member



Alright, I just did a Google with the command:

site:example.com -inurl:www

and it shows about 30 pages of pron and warez links on my site at this URL:

http://example.com.com/examplefolder/wp-content/cache/userlogins/WP/warezand#*$!.html

I also notice a bunch of weird files in that folder on my FTP. Can I just delete all those files? All of the uer logins are people I don't know! Upon further comparison with my other wordpress files, it looks like this entire "cache" directory is completely bogus.

Is there a name for this kind of hack or spam? Whatever it's doing it's killed my bog since late April.

EDIT: after some snooping around my FTP, it looks like my site has been hacked and then somehow redirected. That explains over 100 pages being dropped from non-supps and all my images.

[edited by: Boulder90 at 4:25 am (utc) on May 10, 2009]

g1smd

8:27 am on May 10, 2009 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Make sure all your software is up to date: Apache, PHP, FTP server, Wordpress, etc. Check your .htaccess and .htpasswd files for unauthorised modifications. Clear out all those new files, and change passwords, then speak to your host about the hack. It might be server wide, not just you.

Boulder90

5:50 pm on May 13, 2009 (gmt 0)

5+ Year Member



Thanks g1smd.

Can bad sites linking to you have negative effects? I have two hobby blogs, and I notice that one of them has a bunch of spam pages linking to it. On my major hobby blog (the one referenced in this thread), I have *one* polish/spam site linking to it, but I cannot find my link on that page(even translated, etc)as the #1 referrer to my page. My non-supps are like ten pages now, they were well over 100, asnd the blog had images indexed in Google.

This stinks. If bad links can affect me, I don't know what to do. I can't control that.

Boulder90

5:14 am on May 16, 2009 (gmt 0)

5+ Year Member



An update to anyone bored enough to follow this pathetic story:

Turns out someone hacked my WP uploads folder and inserted this really weird XML.import file that redirected most of my blog to #$@# and pill sites. Google probably didn't like that, lol. They de-indexed all but 4 pages of my site and all it's images. I had well over 100 in the non-supps.

Poof! Good thing it's just a hobby/interest site. I would be even more upset if this was a commercial site that I relied on to pay the bills.

I've searched my blog for other malicious fiels but can't find any. Hopefully deleting that solves the problem.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month