| New? way to trick a spider
|
marten
msg:679252 | 4:32 pm on Jan 11, 2004 (gmt 0) |
I got an idea... not sure it will work.
If I made txt-files run through PHP with an AddType in htaccess i could have a script in my robots.txt that recorded the IP.
Then I would KNOW it was a robot without having to mess with robot-IP-lists.
|
MrSpeed
msg:679253 | 3:59 pm on Jan 12, 2004 (gmt 0) |
That may work but I've noticed a lot of spiders that don't look at robots.txt
|
volatilegx
msg:679254 | 6:28 pm on Jan 15, 2004 (gmt 0) |
Also some browsers and internet accelerators and such look at robots.txt. And, robots don't always access robots.txt prior to accessing other files. Some robots only access robots.txt once a month or so, and continue to request files all month long.
|
|
|
