Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

New? way to trick a spider



4:32 pm on Jan 11, 2004 (gmt 0)

10+ Year Member

I got an idea... not sure it will work.
If I made txt-files run through PHP with an AddType in htaccess i could have a script in my robots.txt that recorded the IP.
Then I would KNOW it was a robot without having to mess with robot-IP-lists.


3:59 pm on Jan 12, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

That may work but I've noticed a lot of spiders that don't look at robots.txt


6:28 pm on Jan 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Also some browsers and internet accelerators and such look at robots.txt. And, robots don't always access robots.txt prior to accessing other files. Some robots only access robots.txt once a month or so, and continue to request files all month long.

Featured Threads

Hot Threads This Week

Hot Threads This Month