Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| NextGenSearchBot 1 Did not obey robots.txt. Is it new? |
6 | cybertime | 2:04 am Nov 21, 2004 | ||||
| Verizon Directory Sales - East Co. |
7 | wilderness | 7:21 pm Nov 19, 2004 | ||||
| Blue Earth Valley |
2 | wilderness | 6:20 am Nov 18, 2004 | ||||
| Jetbot and Travelbot |
7 | Scooter24 | 4:24 am Nov 17, 2004 | ||||
| pipeLiner/0.3a pipeline-search.com |
4 | pendanticist | 5:09 pm Nov 16, 2004 | ||||
| NokodoBot doesn't obey robots.txt and prides itself on the fact |
10 | volatilegx | 6:02 pm Nov 15, 2004 | ||||
| Program Shareware 1.0.0 Can't find *ANY* info on Program Shareware 1.0.0 |
5 | JAB_Creations | 11:48 pm Nov 14, 2004 | ||||
| google shows my old cached site? shows my old landing page, but has been to my site recently from raw logs |
2 | willis1480 | 10:38 pm Nov 13, 2004 | ||||
| Bot list HTTrack or httrack |
2 | wilderness | 10:06 pm Nov 13, 2004 | ||||
| Geona Looks nice.... |
2 | CodeJockey | 9:18 pm Nov 13, 2004 | ||||
| gazz/5.0 Need to update .htaccess file |
6 | guitaristinus | 3:38 pm Nov 9, 2004 | ||||
| \"SeznamBot/1.0\" yes, the correct UA |
3 | bull | 8:59 am Nov 9, 2004 | ||||
| PrassoSunner 1.00 |
2 | bull | 4:42 pm Nov 8, 2004 | ||||
| Nutch from Looksmart |
3 | volatilegx | 10:28 pm Nov 7, 2004 | ||||
| Y!oasis from Yahoo IP |
2 | bull | 7:40 am Nov 7, 2004 | ||||
| Intel Multiple IP's |
2 | wilderness | 4:25 am Nov 7, 2004 | ||||
| bangalore.corp.yahoo.com? Is this a legitimate yahoo spider? global-pix1.bangalore.corp.yahoo.com |
2 | Osian | 12:52 am Nov 6, 2004 | ||||
| 17 | mattie | 2:39 pm Nov 5, 2004 | |||||
| Offline Explorer Not sure where to post this |
2 | chobo321321 | 12:48 am Nov 3, 2004 | ||||
| ExoticCrawler ban it? |
4 | guitaristinus | 10:44 pm Nov 2, 2004 | ||||
| new private search engine? jeteye.com[2] ( 1 2 ) |
43 | macrost | 5:27 am Nov 2, 2004 | ||||
| innerprise.net? |
2 | blaketar | 4:05 am Nov 2, 2004 | ||||
| Amazon |
2 | wilderness | 1:10 am Oct 29, 2004 | ||||
| Jakarta Commons-HttpClient/2.0rc2 Anyone banning it? |
2 | guitaristinus | 2:10 pm Oct 28, 2004 | ||||
| WebRescuer v0.2.4 Anyone know what this is? |
3 | Busynut | 11:45 am Oct 28, 2004 |
