Welcome to WebmasterWorld Guest from 54.167.174.11

Forum Moderators: open

Message Too Old, No Replies

New? way to trick a spider

   
4:32 pm on Jan 11, 2004 (gmt 0)

10+ Year Member



I got an idea... not sure it will work.
If I made txt-files run through PHP with an AddType in htaccess i could have a script in my robots.txt that recorded the IP.
Then I would KNOW it was a robot without having to mess with robot-IP-lists.
3:59 pm on Jan 12, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That may work but I've noticed a lot of spiders that don't look at robots.txt
6:28 pm on Jan 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Also some browsers and internet accelerators and such look at robots.txt. And, robots don't always access robots.txt prior to accessing other files. Some robots only access robots.txt once a month or so, and continue to request files all month long.