Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
It spiders repeatedly the same page with differente session ids:
18.104.22.168 - - [20/Feb/2002:04:01:59 +0100] "GET /index.php3?PHPSESSID=2ab18fdd6ca1320f644422b44daffa2d HTTP/1.0" 200 59126 "-" "FAST-WebCrawler/3.3 (firstname.lastname@example.org; [fast.no...] "-"
22.214.171.124 - - [20/Feb/2002:04:02:24 +0100] "GET /index.php3?PHPSESSID=768e7265ce0735e14ff41532e05ea0e9 HTTP/1.0" 200 59126 "-" "FAST-WebCrawler/3.3 (email@example.com; [fast.no...] "-"
Is there some simple solution how to avoid this? We get many thousands hits for these spurious urls from the fastbot...
Don't write session ID's in the url. :)
A solution where the session ID is stored in a cookie works much better.
The other option would be to check IP addresses before writing the session ID's to the URL. If a search engine IP is detected, the server simply serves the page without the ID. All IP's not on your list get the unique ID.