Welcome to WebmasterWorld Guest from 50.19.135.67

Forum Moderators: open

Message Too Old, No Replies

New? way to trick a spider

     

marten

4:32 pm on Jan 11, 2004 (gmt 0)

10+ Year Member



I got an idea... not sure it will work.
If I made txt-files run through PHP with an AddType in htaccess i could have a script in my robots.txt that recorded the IP.
Then I would KNOW it was a robot without having to mess with robot-IP-lists.

MrSpeed

3:59 pm on Jan 12, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That may work but I've noticed a lot of spiders that don't look at robots.txt

volatilegx

6:28 pm on Jan 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Also some browsers and internet accelerators and such look at robots.txt. And, robots don't always access robots.txt prior to accessing other files. Some robots only access robots.txt once a month or so, and continue to request files all month long.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month