Welcome to WebmasterWorld Guest from 54.81.116.187
Preferred Member
joined:Jan 19, 2004
posts:562
votes: 0
Referencing back to a great post on a PHP spider trap [webmasterworld.com] here about 18 months ago.
Boy I have had fun with this one on five sites. The trap sends me an email when a bad bot gobbles my /trap.php. It's been working with out problems for over a year, and I'm not sure it has a problem even now. I think it's with MSN.
Here's the deal. I was going over the pages indexed by the big three for one of my sites and there was the surprise. MSN Search had neatly indexed mydomain.com/trap.php A great way for anyone who happened to stumble on it to block themselves.
I checked the log file and MSN's bot never touched that trap.php or banned itself. I checked my sitemap.xml to see if I had stuck it in there by accident. Is it possible that it grabbed that from my server log as it would have been put there when testing?
The site is only a couple of weeks old and just starting to get spidered. For some reason Google is letting the others get ahead on this one. If anyone has an idea I would sure enjoy hearing it.
[edited by: jatar_k at 5:54 am (utc) on Feb. 2, 2006]
[edit reason] fixed link [/edit]
Administrator
joined:July 31, 2003
posts:12548
votes: 2
No, I don't know how it could ever get to your server log. I would guess that at some point the url was available and it picked it up. I'm surprised you can't find the hit in your access log.
Preferred Member
joined:Jan 19, 2004
posts:562
votes: 0
I would guess that at some point the url was available and it picked it up
My thoughts exactly but where? I really looked hard in the log. I used awstats with the rawlog plugin. It has a search that would have picked it up.
In MSN's index it was listed with just the URL as the title link and of course no link to cached page or description. Very strange.
Full Member
joined:Dec 1, 2003
posts:311
votes: 0
If what you see is the URL of your trap page in the index, it is a normal behavior for msn. It happens to me too. What happens is that the bot will pick up the link from any page which links to your trap but following the rules in your robots.txt it won't hit the page. However, it will be listed in the index as URL only.
In fact, for my site, when I do a site:example.com command using msn, it comes in the top 5 lol!.
Preferred Member
joined:Jan 19, 2004
posts:562
votes: 0
caspita, Well I'm glad it's not just me. I thought about trying to change something, but in reality who will ever find that link in MSN except me.
Join The Conversation
- Register For Free! - Become a Pro Member!
- See forum categories - Enter the Forum
Moderators and Top Contributors
- Moderator List | Top Contributors:This Week, This Month, Mar, Feb, Archive, Top 100 All Time, Top Voted Members
Hot Threads This Week
- Google Updates and SERP Changes - April 2018
- April 2018 AdSense Earnings & Observations
- Google AdSense: Expands "Funding Choices" To Replace Lost Revenue From Ad Blocking
- What Makes Good Web Content Today
- MauiBot
- EU GDPR (General Data Protection Regulation) and Analytics cookies
- Mozilla: Internet Health Report 2018
- Facebook Goes for World-Wide ToS Changes
- Google Updates and SERP Changes - March 2018
- Chrome 66 for Windows, Mac and Linux Rolls Out
Featured Threads
- Facebook Goes for World-Wide ToS Changes
- Chrome 66 for Windows, Mac and Linux Rolls Out
- Google AdSense: Expands "Funding Choices" To Replace Lost Revenue From Ad Blocking
- What Makes Good Web Content Today
- Mozilla: Internet Health Report 2018
- Facebook Launches Data Abuse Bounty
- How About an RSS Revival
- Webmasters: Distrust of Symantec PKI in Chrome 66
- A Directory is a Utility far more than it is a Business Model
- Mozilla Readies Virtual and Augmented Reality Browser, Firefox Reality