homepage Welcome to WebmasterWorld Guest from 54.196.197.153
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / Cloaking
Forum Library, Charter, Moderator: open

Cloaking Forum

    
New? way to trick a spider
marten




msg:679252
 4:32 pm on Jan 11, 2004 (gmt 0)

I got an idea... not sure it will work.
If I made txt-files run through PHP with an AddType in htaccess i could have a script in my robots.txt that recorded the IP.
Then I would KNOW it was a robot without having to mess with robot-IP-lists.

 

MrSpeed




msg:679253
 3:59 pm on Jan 12, 2004 (gmt 0)

That may work but I've noticed a lot of spiders that don't look at robots.txt

volatilegx




msg:679254
 6:28 pm on Jan 15, 2004 (gmt 0)

Also some browsers and internet accelerators and such look at robots.txt. And, robots don't always access robots.txt prior to accessing other files. Some robots only access robots.txt once a month or so, and continue to request files all month long.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Cloaking
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved