Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| This is new... Ink UK? |
2 | toolman | 9:54 am June 19, 2001 | ||||
| Anyone know who this is? 209.67.206.133 |
3 | toolman | 8:16 am June 17, 2001 | ||||
| MSN what is the useragent |
2 | zipup | 9:02 pm June 15, 2001 | ||||
| List your banned IP's / domains and why |
7 | icehousedesigns | 1:06 pm June 14, 2001 | ||||
| Teoma Spider IP list where can I find one? |
5 | Travoli | 1:04 pm June 14, 2001 | ||||
| Crescent Internet ToolPak HTTP OLE Control v.1.0 Should I ban it in my robots.txt file? |
2 | merlin78 | 5:26 am June 14, 2001 | ||||
| Inktomi hitting pages over and over what the heck? |
5 | Travoli | 1:04 pm June 13, 2001 | ||||
| AOL UAbut different IP |
4 | toolman | 8:08 pm June 12, 2001 | ||||
| Site friendly spiders : the list |
4 | Brett_Tabke | 7:56 pm June 12, 2001 | ||||
| A new UA to me Fast |
2 | startup | 7:54 pm June 11, 2001 | ||||
| robot names - please help |
5 | Ade | 6:37 pm June 11, 2001 | ||||
| Spider Spammer Spammin' the UA |
3 | awoyo | 12:47 pm June 11, 2001 | ||||
| Anyone know what this is? larbin_2.1.0 larbin2.1.0@somewhere.com |
3 | icehousedesigns | 6:06 pm June 7, 2001 | ||||
| New 404 test from DIIbot |
2 | jimmykav | 6:04 pm June 7, 2001 | ||||
| MSIEcrawler For those of you that didn't know... |
3 | icehousedesigns | 3:35 pm June 7, 2001 | ||||
| 2 | Froggyman | 10:24 am June 7, 2001 | |||||
| Another one for the experts. widevine.seattle.winstar.net |
3 | icehousedesigns | 7:40 pm June 6, 2001 | ||||
| AOL spider? ACB998C0.ipt.aol.com |
2 | icehousedesigns | 1:52 pm June 6, 2001 | ||||
| Heads Up! |
9 | toolman | 3:08 pm June 5, 2001 | ||||
| inktomi2-lut.server.ntl.com .. Found this one grabbing only one page. |
2 | Rumbas | 2:28 pm June 5, 2001 | ||||
| psbot/0.1 picsearch is currently indexing pictures |
8 | Son_House | 12:32 pm June 5, 2001 | ||||
| 216.35.112.143 is Fast ? |
10 | berno | 3:19 am June 5, 2001 | ||||
| Why run a 404 test? If they're searching for stolen content? |
6 | mivox | 12:33 am June 5, 2001 | ||||
| MIIxpc? |
10 | mark_roach | 12:03 am June 5, 2001 | ||||
| New Altavista Behaviour... libwww-perl/5.46 for av? |
5 | skirril | 11:51 pm June 4, 2001 |
