Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| PingALing ANOTHER "is your website up" bot! |
4 | carfac | 5:26 pm Oct 14, 2002 | ||||
| NG/1.0 update Spider from France reads, ignores robots.txt |
13 | jdMorgan | 5:08 pm Oct 14, 2002 | ||||
| webcrawler@nospam |
4 | Ove | 3:44 pm Oct 14, 2002 | ||||
| GirafaBot Who? What? Where? |
8 | frontpage | 10:39 am Oct 14, 2002 | ||||
| URL Spider Pro |
7 | misosoph | 10:58 pm Oct 13, 2002 | ||||
| Interesting visit by AskJeeves Making a dead page cleanup pass? |
5 | jdMorgan | 7:17 pm Oct 13, 2002 | ||||
| Why is important to know which spiders are visiting? |
7 | dauction | 3:42 am Oct 13, 2002 | ||||
| Mozilla/2.0 |
2 | Scooter24 | 12:23 pm Oct 11, 2002 | ||||
| Lexibot Allow or ban |
4 | mack | 11:38 pm Oct 10, 2002 | ||||
| rabaz Seems well behaved |
2 | carfac | 5:19 pm Oct 10, 2002 | ||||
| ua: ht |
3 | carfac | 4:50 pm Oct 10, 2002 | ||||
| What is contype UA? |
5 | transistor | 3:55 pm Oct 10, 2002 | ||||
| What is Alexa? |
3 | palmpal | 5:45 am Oct 10, 2002 | ||||
| YComp 5.x.x.x in so many user agents what the heck is it? |
10 | GaryK | 11:25 pm Oct 6, 2002 | ||||
| Ywo new- and different- ones spotted today |
6 | carfac | 7:46 pm Oct 6, 2002 | ||||
| Microsoft Internet Explorer/4.40.426 Obvious - yet very strange - fake browser UA |
5 | Dreamquick | 6:10 pm Oct 4, 2002 | ||||
| SlySearch? |
22 | hanuman | 3:21 pm Oct 4, 2002 | ||||
| $botname Someone loose a variable? |
3 | carfac | 2:18 pm Oct 4, 2002 | ||||
| MARTINI - sysadmin@looksmart.net Is this a new L$ robot or just someone who likes their liquor? |
10 | Dreamquick | 7:58 am Oct 4, 2002 | ||||
| HTTrack 3.0x what is this? |
5 | allybongo | 8:48 pm Oct 3, 2002 | ||||
| Whois "-" "-"? How do I block "-" "-" from my Robots.txt file? |
11 | jrota | 5:35 pm Oct 3, 2002 | ||||
| Need a Bit of a Tutorial... Trying to figure out what it is... |
3 | carfac | 8:04 pm Oct 2, 2002 | ||||
| Web Download Tool Something else to ban |
12 | carfac | 6:39 pm Oct 2, 2002 | ||||
| We weirdest UA Ever! |
5 | carfac | 7:23 pm Oct 1, 2002 | ||||
| New Fast IP |
3 | volatilegx | 5:12 pm Oct 1, 2002 |
