Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
It spiders repeatedly the same page with differente session ids:
184.108.40.206 - - [20/Feb/2002:04:01:59 +0100] "GET /index.php3?PHPSESSID=2ab18fdd6ca1320f644422b44daffa2d HTTP/1.0" 200 59126 "-" "FAST-WebCrawler/3.3 (email@example.com; [fast.no...] "-"
220.127.116.11 - - [20/Feb/2002:04:02:24 +0100] "GET /index.php3?PHPSESSID=768e7265ce0735e14ff41532e05ea0e9 HTTP/1.0" 200 59126 "-" "FAST-WebCrawler/3.3 (firstname.lastname@example.org; [fast.no...] "-"
Is there some simple solution how to avoid this? We get many thousands hits for these spurious urls from the fastbot...
Don't write session ID's in the url. :)
A solution where the session ID is stored in a cookie works much better.
The other option would be to check IP addresses before writing the session ID's to the URL. If a search engine IP is detected, the server simply serves the page without the ID. All IP's not on your list get the unique ID.
You could probably even handle it just using the UA.