homepage Welcome to WebmasterWorld Guest from 54.204.58.87
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / Cloaking
Forum Library, Charter, Moderator: open

Cloaking Forum

    
Yahoo is GIving me FITS over the last few months!
Am I missing a new UA or IP block? Algo change?
Friday




msg:675361
 11:47 pm on Aug 3, 2005 (gmt 0)

Of all the sites I manage SEO for, five of them utilizie a very good (If I do say so myself), proprietary IP cloaking system. It even e-mails me whenever a known UA visits one of my sites from an IP address I don't currently have in my IP DB (so I can research it and either add it or ban it).

Most of these (few) cloaking clients have been clients for several years and ALL of them have been dominating the SERPs at Yahoo! (and others) for their respective Twelve (12) keywords/phrases for as long as I've been working with them.

BUT.... Within the last 2-3 months they are dropping like FLIES at Yahoo!

This is NOT good for my cash flow situation, since I charge by perfomance -- no performance, no charge for that month (base monthly fee once minimum monthly requirements have been met then bonuses for each #1, #2-5, #6-10, and #11-20 spots achieved).
(With a monthly CAP in place, of course, to keep their bills from running into the tens-of-thousands a month).

It;'s a VERY GOOD Gig.
Life is very very good!
:-)
But I'm getting nervous.

Anyone else experienced this "problem" with Yahoo! within the last few months?

The sites in question have not been penalized or banned, and all clients remain in the Yahoo! index. They've just dropped 50-60 spots! (OUCH!).

Either a major algorithm change has rendered my (Yahoo!) optiumization formula obsolete OR (and I suspect this), I've missed a new Yahoo! User_Agent and/or string of IP Addrsses.

The UA's I currently track are:

$known_ua = "alltheweb¦rdcrawler¦uswal¦ATW¦Fast PartnerSite Crawler¦FAST Enterprise Crawler¦FAST FirstPage¦
FAST-WebCrawler¦Robozilla¦Safegate¦overture¦RealWebCrawler¦sac2¦fastsearch¦msn-search¦exava¦CrawlX¦TurnItInBot¦
inktomisearch¦Fast-WebCrawler¦overture¦datafast¦translate¦Mediapartner¦Wisenut¦Googlebot¦Fast¦Slurp¦Architext¦
Scooter¦Sidewinder¦T-Rex¦sv\.av¦Lycos¦Vista¦Webcrawler¦Jeeves¦Teoma¦Excite¦MSNBOT¦modspider¦AVSearch¦BSDSeek¦
Inktomi¦libwww-perl\/5\.52¦Mercator¦av\.com¦Merc_resh¦UltraSeek¦MantraAgent¦grabber¦InfoSeek¦DirectHit¦D-H¦
Direct Hit¦DH¦Looksmart¦AlltheWeb¦MARTINI¦Zealbot¦Zyborg¦Seeker¦MnogoSearch¦MongoSearch¦Wget¦Look\.com¦
SEARCHALOT¦MMCrawler¦YahooSeeker¦Yahoo-Seeker¦atw-crawler";

The IPS I currently have flagged as valid are:

$yahoo =
"^68.142.230.181¦^66.228.164.108¦^202.165.98.144¦^202.165.98.141¦^202.165.98.143¦^202.165.98.146¦^202.165.98.142¦
^202.165.98.139¦^66.228.164.90¦^68.142.230.194¦^68.142.231.49¦^66.228.164.115¦^66.163.170.157¦^66.163.170.178¦
^66.228.164.201¦^68.142.230.202¦^66.196.91¦^66.196.90¦^66.196.91¦^68.142.249¦^68.142.250¦^68.142.251¦^66.228.165.31¦
^66.228.164.203¦^64.69.39.71¦^66.196.65.34¦^66.196.65.37¦^66.196.65.39¦^66.196.65.42¦^66.196.65.49¦^66.196.65.50¦
^66.196.65.51¦^66.196.65.52¦^66.196.65.53¦^66.196.65.54¦^66.196.65.55¦^66.196.65.56¦^66.196.65.57¦^66.196.65.58¦
^66.228.164.119¦^66.228.164.102¦^66.228.164.149¦^68.142.230.195¦^68.142.230.197¦^68.142.230.179¦^66.163.170.176¦
^68.142.230.190¦^68.142.230.179¦^68.142.230.184¦^68.142.230.193¦^68.142.249¦^68.142.250¦^68.142.251¦^66.196.80.112¦
^66.196.80.120¦^66.196.80.98¦^66.196.101.98¦^66.196.80.56¦^66.196.80.72¦^66.196.80.83¦^66.196.80.132¦^66.163.170.190¦
^66.196.80.14¦^66.196.80.34¦^66.196.80.57¦^66.196.80.51¦66.196.80.79¦^66.196.80.10¦^66.196.80.100¦^66.196.80.101¦
^66.196.80.102¦^66.196.80.103¦^66.196.80.104¦^66.196.80.105¦^66.196.80.106¦^66.196.80.107¦^66.196.80.108¦
^66.196.80.109¦^66.196.80.110¦^66.196.80.111¦^66.196.80.114¦^66.196.80.115¦^66.196.80.116¦^66.196.80.118¦
^66.196.80.119¦^66.196.80.121¦^66.196.80.122¦^66.196.80.124¦^66.196.80.126¦^66.196.80.128¦^66.196.80.129¦
^66.196.80.13¦^66.196.80.130¦^66.196.80.132¦^66.196.80.134¦^66.196.80.135¦^66.196.80.136¦^66.196.80.138¦
^66.196.80.139¦^66.196.80.140¦^66.196.80.142¦^66.196.80.143¦^66.196.80.144¦^66.196.80.145¦^66.196.80.146¦
^66.196.80.147¦^66.196.80.148¦^66.196.80.15¦^66.196.80.152¦^66.196.80.154¦^66.196.80.155¦^66.196.80.156¦
^66.196.80.16¦^66.196.80.160¦^66.196.80.164¦^66.196.80.165¦^66.196.80.17¦^66.196.80.18¦^66.196.80.19¦^66.196.80.20¦
^66.196.80.21¦^66.196.80.23¦^66.196.80.24¦^66.196.80.25¦^66.196.80.26¦^66.196.80.27¦^66.196.80.29¦^66.196.80.30¦
^66.196.80.31¦^66.196.80.32¦^66.196.80.36¦^66.196.80.37¦^66.196.80.38¦^66.196.80.40¦^66.196.80.42¦^66.196.80.43¦
^66.196.80.44¦^66.196.80.45¦^66.196.80.46¦^66.196.80.47¦^66.196.80.48¦^66.196.80.49¦^66.196.80.51¦^66.196.80.53¦
^66.196.80.54¦^66.196.80.55¦^66.196.80.61¦^66.196.80.62¦^66.196.80.63¦^66.196.80.64¦^66.196.80.65¦^66.196.80.66¦
^66.196.80.67¦^66.196.80.69¦^66.196.80.68¦^66.196.80.70¦^66.196.80.71¦^66.196.80.73¦^66.196.80.74¦^66.196.80.76¦
^66.196.80.77¦^66.196.80.79¦^66.196.80.81¦^66.196.80.82¦^66.196.80.84¦^66.196.80.85¦^66.196.80.86¦^66.196.80.87¦
^66.196.80.88¦^66.196.80.89¦^66.196.80.90¦^66.196.80.91¦^66.196.80.92¦^66.196.80.93¦^66.196.80.94¦^66.196.80.95¦
^66.196.80.96¦^66.196.80.97¦^66.196.80.98¦^66.196.80.99¦^66.196.91.8¦^66.196.91.9¦^66.196.92¦^216.109.126.147¦
^216.109.126.137¦^216.109.126.160¦^216.109.126.141¦^66.163.170.179¦^66.196.101.153¦^216.109.126.146¦^216.109.126.158¦
^216.109.126.152¦^66.163.170.166¦^216.109.126.133¦^216.109.126.138¦^216.109.126.150¦^216.109.126.161¦^216.109.126.143¦
^216.109.126.139¦^66.196.101.97¦^66.94.233.90¦^216.109.126.131¦^216.109.126.157¦^216.109.126.159¦^216.109.126.145¦
^66.163.170.167¦^66.196.67.108¦^66.94.233.84¦^66.196.91.11¦^66.196.91.118¦^66.196.91.117¦^66.196.91.115¦^66.196.91.111¦
^66.196.91.112¦^66.196.91.100¦^66.196.91.101¦^66.196.91.102¦^66.196.91.105¦^66.196.91.21¦^66.196.91.31¦^66.196.91.32¦
^66.196.91.34¦^66.196.91.37¦^66.196.91.40¦^66.196.91.46¦^66.196.91.50¦^66.196.91.53¦^66.196.91.75¦^66.196.91.77¦
^66.196.91.78¦^66.196.91.79¦^66.196.91.80¦^66.196.91.81¦^66.196.91.88¦^66.196.91.96¦^66.196.91.97¦^66.196.91.100¦
^66.196.91.110¦^66.196.91.113¦^66.196.91.103¦^66.196.91.104¦^66.196.91.106¦^66.196.91.107¦^66.196.91.108¦^66.196.91.109¦
^66.196.91.18¦^66.196.91.19¦^66.196.91.20¦^66.196.91.22¦^66.196.91.23¦^66.196.91.24¦^66.196.91.25¦^66.196.91.26¦
^66.196.91.27¦^66.196.91.28¦^66.196.91.29¦^66.196.91.30¦^66.196.91.33¦^66.196.91.35¦^66.196.91.36¦^66.196.91.38¦
^66.196.91.39¦^66.196.91.41¦^66.196.91.42¦^66.196.91.43¦^66.196.91.44¦^66.196.91.45¦^66.196.91.47¦^66.196.91.48¦
^66.196.91.49¦^66.196.91.51¦^66.196.91.52¦^66.196.91.54¦^66.196.91.55¦^66.196.91.56¦^66.196.91.57¦^66.196.91.58¦
^66.196.91.59¦^66.196.91.60¦^66.196.91.61¦^66.196.91.62¦^66.196.91.63¦^66.196.91.64¦^66.196.91.65¦^66.196.91.66¦
^66.196.91.67¦^66.196.91.68¦^66.196.91.69¦^66.196.91.70¦^66.196.91.71¦^66.196.91.72¦^66.196.91.73¦^66.196.91.74¦
^66.196.91.76¦^66.196.91.82¦^66.196.91.83¦^66.196.91.84¦^66.196.91.86¦^66.196.91.85¦^66.196.91.87¦^66.196.91.89¦
^66.196.91.90¦^66.196.91.91¦^66.196.91.92¦^66.196.91.93¦^66.196.91.94¦^66.196.91.95¦^66.196.91.98¦^66.196.91.99¦
^66.196.101.91¦^66.196.91.17¦^66.196.101.85¦^66.196.91.15¦^66.196.91.14¦^66.196.91.16¦^66.196.91.12¦^66.94.232.246¦
^66.196.101.87¦^66.163.170.170¦^66.196.101.96¦^66.163.170.161¦^66.196.101.89¦^66.196.67.101¦^66.196.101.94¦
^66.163.170.193¦^66.163.170.159¦^66.163.170.185¦^66.163.170.184¦^66.196.101.84¦^66.196.101.83¦^66.163.170.192¦
^66.196.101.92¦^66.196.101.93¦^66.196.101.86¦^66.196.101.90¦^66.196.101.88¦^66.163.170.180¦^66.163.170.172¦
^66.77.73.186¦^66.196.93.7¦^66.196.93.24¦^66.196.90¦^66.196.93.7¦^66.196.93.24¦^66.196.93.6";

And, just in case, some old partner IPs too:

$infoseek =
"^210.148.160.157¦^211.18.214.194¦^212.185.44.10¦^212.185.44.11¦^212.185.44.12¦^212.185.44.15¦^210.236.233.131¦
^210.236.233.132¦^210.236.233.133¦^210.236.233.135¦^210.236.233.136¦^210.236.233.137¦^210.236.233.139¦^210.236.233.150¦
^210.236.233.151¦^210.236.233.160¦^210.236.233.161¦^205.226.203.186¦^205.226.203.35¦^205.226.203.56¦^205.226.203.62¦
^204.202.132.19¦^202.33.250.146¦^202.33.250.147¦^202.33.250.148¦^202.33.250.149¦^202.33.250.150¦^202.33.250.151¦
^202.33.250.152¦^202.33.250.153¦^202.33.250.154¦^195.145.119.24¦^195.145.119.25¦^204.162.96.0¦^204.162.99.0¦
^204.162.96¦^204.162.97¦^204.162.98¦^204.162.99¦^198.5.208.100¦^198.5.208¦^198.5.210¦^204.162.96.124¦^204.162.97.17¦
^204.162.98.11¦^205.226.201.30¦^205.226.203.35¦^205.226.204.238¦^205.226.204¦^210.236.233.155¦^210.236.233";

$altavista = "^216.39.50¦^66.17.148.128¦^66.17.148.129¦^66.17.148.130¦^66.17.148.131¦^66.17.148.132¦^66.17.148.133¦
^66.17.148.134¦^66.17.148.135¦^66.17.148.136¦^66.17.148.137¦^66.17.148.138¦^66.17.148.139¦^66.17.148.140¦
^66.17.148.141¦^66.17.148.142¦^66.17.148.143¦^66.17.148.144¦^66.17.148.145¦^66.17.148.146¦^66.17.148.147¦
^66.17.148.148¦^66.17.148.149¦^66.17.148.150¦^66.17.148.151¦^66.17.148.152¦^66.17.148.153¦^66.17.148.154¦
^66.17.148.155¦^66.17.148.156¦^66.17.148.157¦^66.17.148.158¦^66.17.148.159¦^66.17.148.160¦^66.17.148.161¦
^66.17.148.162¦^66.17.148.163¦^66.17.148.164¦^66.17.148.165¦^66.17.148.166¦^66.17.148.167¦^66.17.148.168¦
^66.17.148.169¦^66.17.148.170¦^66.17.148.171¦^66.17.148.172¦^66.17.148.173¦^66.17.148.174¦^66.17.148.175¦
^66.17.148.176¦^66.17.148.177¦^66.17.148.178¦^66.17.148.179¦^66.17.148.180¦^66.17.148.181¦^66.17.148.182¦
^66.17.148.183¦^66.17.148.184¦^66.17.148.185¦^66.17.148.186¦^66.17.148.187¦^66.17.148.188¦^66.17.148.189¦
^66.17.148.190¦^66.17.148.191¦^216.39.48¦^208.185.243.148¦^216.198.200.119¦^212.187.213.171¦^212.187.213.172¦
^212.187.213.173¦^212.187.213.174^¦212.187.213.175¦^209.73.174.250¦^209.73.174.251¦^209.247.40.246¦^208.221.35.200¦
^208.221.35.201¦^208.221.35.202¦^208.221.35.203¦^208.221.35.204¦^208.221.35.205¦^208.221.35.206¦^208.221.35.207¦
^205.229.83.18¦^194.221.84.11¦^194.221.84.15¦^194.221.84.31¦^194.221.84.32¦^194.221.84.33¦^194.221.84.34¦
^194.221.84.38¦^194.221.84.39¦^194.221.84.40¦^194.221.84.41¦^194.51.33.72¦^194.201.146.1¦^194.201.146.24¦
^194.112.94.251¦^194.112.94.252¦^194.112.94.250¦^128.177.244.86¦^128.177.244.100¦^212.187.227¦^209.73.191¦
^209.73.160¦^204.123.13¦^208.221.32¦^204.123.28¦^204.123.2¦^204.123.3¦^204.123.4¦^204.123.5¦^204.123.6¦^204.123.7¦
^204.123.8¦^204.123.10¦^204.123.11¦^204.123.12¦^64.75.36¦^204.123.0.0¦^204.123.255.0¦^204.74.103.39¦^204.123.9¦
^64.152.75¦^209.73.162¦^209.73.164¦^204.152.191¦^204.152.190¦^128.177.243¦^212.187.226";

I _KNOW_ some of these IPs are outdated and I'm in the process going through my entire DB and validating and/or weeding out old ones that have been sold/transferrred.

This is KILLING My Credibility
And endangering my livelihood!
Plus, I genuinely feel a sincere duty to my clients -- they are good people selling good, needed products and services. And they;re all hionest, straight-shooters (or they would NOT be clients).

Best Regards,
Friday

[edited by: volatilegx at 3:03 pm (utc) on Aug. 4, 2005]
[edit reason] fixed horizontal scrolling problem [/edit]

 

volatilegx




msg:675362
 3:05 pm on Aug 4, 2005 (gmt 0)

I suspect the recent Yahoo algo changes may have something to do with it. The evidence is that the pages are still in the index.

robbyboo




msg:675363
 10:40 pm on Aug 11, 2005 (gmt 0)

Yahoo is under alot of pressure both, from a legal standpoint and a server standpoint. Things will not go right until they get it all straightened out. So, we wait and wonder where the hell Waldo REALLY is.

Friday




msg:675364
 3:44 pm on Aug 20, 2005 (gmt 0)

Thanks Rankboy.

But I use my own series of PHP scripts to track spider activity and IP Addresses in real time.

Whenever anyone/thing visits one of my sites, the script grabs the user_agent and IP Address and compares them against my lists of known Search Engine User_Agents and IP Addresses.

If it's a known Search Engine UA BUT with an IP Address that's NOT vurrently in my database of SE IPs, the server sends me an e-mail Alert (via PHPs built-in "mail()" function)... something like follows:

#################
Date: Sat, 20 Aug 2005 10:16:40 -0500 (CDT)
From: SpiderWatch@#*$!#*$!xxxxxxxx.com
Subject: Spider Alert - 207.68.146.68-msnbot/1.0 (+http://search.msn.com/msnbot.htm)
To: xxxxxxxxxxx@twcny.rr.com
X-Virus-Scanned: Symantec AntiVirus Scan Engine
Original-recipient: rfc822;xxxxxxxxxxxxx@twcny.rr.com

The Spider "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"
APPARENTLY has a NEW IP address: 207.68.146.68
It visited the page: xxxxx.php at :
Sat, 20 Aug 2005 10:16:40 -0500 (CDT)
###################################

THEN, all I need to do is a quick "WHOIS" to validate it. If it's from a new netrange, I not that for future reference.

If it's just another jerkoff looking to see if I'm stupid enought to cloak by User_Agent, I ban his/her IP from the site in question via ".htaccess."
Some are so stupid they sign up for HotMail accounts and visit from a MS IP Address, but they're still definitely not "MSNBot."
8-P

It's a quaint little system, but it works for me.

Best Regards,
Friday

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Cloaking
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved