Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Zeus and Sqworm |
3 | coyote | 6:53 pm Aug 19, 2003 | ||||
| Getty Images (206.28.73.1) ignoring robots.txt Fake user-agent, tries to grab all pages |
3 | jazzguy | 5:01 pm Aug 19, 2003 | ||||
| Good robots and bad ones - your opinion. script to help get away bad spiders from our sites. |
3 | AlexPar | 6:22 pm Aug 18, 2003 | ||||
| UA: Java/1.4.1_03 |
2 | coyote | 4:23 pm Aug 18, 2003 | ||||
| SE's out in force |
4 | Brett_Tabke | 3:51 pm Aug 18, 2003 | ||||
| Anyone recognise 80.58.6.170 |
4 | decstar | 10:08 am Aug 18, 2003 | ||||
| new directory spider 64.69.79.210 |
3 | Peeress | 1:45 pm Aug 17, 2003 | ||||
| New bot? Dmoz-checker? |
4 | claus | 6:45 am Aug 17, 2003 | ||||
| QuepasaCreep anyone have a clue who this is |
2 | penfold25 | 7:03 pm Aug 16, 2003 | ||||
| Need to Block Pesky Spider markmonitor |
9 | guillermo5000 | 10:19 pm Aug 15, 2003 | ||||
| who am i yet another bot |
6 | coyote | 9:39 pm Aug 15, 2003 | ||||
| Someone repeatedly asking for two images that don't exist They have been hitting me hard requesting them |
7 | AWildman | 8:52 pm Aug 15, 2003 | ||||
| Deciphering log files tracking bots |
4 | Mr_Busby | 4:50 pm Aug 15, 2003 | ||||
| Google? Not sure. |
3 | wilderness | 3:12 pm Aug 15, 2003 | ||||
| spider.ilab.sztaki.hu Seems well behaved |
2 | WitchLars | 3:06 pm Aug 15, 2003 | ||||
| Multiple IP Access Who, What, How? Bot? |
5 | Latigo | 3:35 pm Aug 14, 2003 | ||||
| Pita now is called WebVac name change |
6 | webvaccrawler | 4:24 pm Aug 13, 2003 | ||||
| New UA: FavOrg Favicon/link-checker |
2 | claus | 10:04 pm Aug 12, 2003 | ||||
| minibot(NaverRobot)/1.0 Who is this? |
16 | darryl_foshee | 6:24 pm Aug 12, 2003 | ||||
| NaverRobot masquerading as GoogleBot I don't like it |
6 | Powdork | 3:31 pm Aug 12, 2003 | ||||
| ia_archiver archive.org |
10 | wilderness | 3:08 am Aug 12, 2003 | ||||
| Kasetsart University (Department of Computer Engineering) in Bangkok, SpiderKU/0.9 |
3 | aaron2b | 6:00 pm Aug 11, 2003 | ||||
| What is readwebpage? |
2 | BlueSky | 4:30 pm Aug 11, 2003 | ||||
| InternetLinkAgent/3.1 |
4 | rainborick | 1:48 pm Aug 11, 2003 | ||||
| Who or what is WebHiker/1.0 |
3 | viggen | 12:38 pm Aug 11, 2003 |
