Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Writing a bot for my site to check links. Is robots.txt validation required? |
18 | Duskrider | 6:52 am June 23, 2006 | ||||
| Gigabot UA change? dropped versioning |
5 | jdMorgan | 6:45 am June 23, 2006 | ||||
| Announcement: TrueLocal TUCKER 0.1 Mozilla/5.0 (compatible; TUCKER/0.1; +http://www.truelocal.com/tucker.aspx) |
4 | bakedjake | 9:24 pm June 22, 2006 | ||||
| Mozilla (Google Web Accelerator Cache Warmer; Google-TR-1) IP belongs to Road Runner |
9 | GaryK | 7:27 pm June 22, 2006 | ||||
| "mail.visvo.com" runs trio o' bots: Anonymous; NutchCVS; Skywalker TLD "visvo.com" nameservers = .yahoo.com (!) |
8 | Pfui | 5:56 pm June 22, 2006 | ||||
| EmeraldShield crawled a disallowed folder |
13 | Mokita | 2:18 am June 22, 2006 | ||||
| Spider tracking Tools to analyse crawling |
3 | rupalis | 12:26 am June 22, 2006 | ||||
| Bot coming from Everyones Internet |
9 | Mokita | 12:00 am June 22, 2006 | ||||
| User Agent: Ken |
2 | fusion5 | 8:34 pm June 20, 2006 | ||||
| FDSE robot |
5 | bull | 1:18 am June 20, 2006 | ||||
| Snappy/1.1 urltrends bot |
4 | Mokita | 6:04 am June 19, 2006 | ||||
| Nutch Sightings from 100+ IPs |
18 | incrediBILL | 3:51 am June 19, 2006 | ||||
| I'm a newbie Know what this is? |
5 | jcmoon | 5:03 pm June 15, 2006 | ||||
| what's wrong with my sitemap.xml sitemap |
4 | youriv | 10:56 pm June 14, 2006 | ||||
| WISEbot/1.0 (WISEbot@wisenut.co.kr; http://wisebot.wisenut.co.kr) No robots.txt |
12 | GaryK | 11:26 pm June 12, 2006 | ||||
| exactseek.com Last forum thread seems to be from 2002 |
5 | GaryK | 6:09 pm June 12, 2006 | ||||
| Teoma's new IPs a direct hit... |
14 | incrediBILL | 11:32 pm June 11, 2006 | ||||
| Worio don't worio, be happy |
5 | incrediBILL | 8:26 pm June 11, 2006 | ||||
| Yahoo? Overture? mozilla/4.0 |
10 | fiestagirl | 12:54 pm June 11, 2006 | ||||
| Looksmart crawling with Mozilla/4.0 User-agent Doesn't identify itself |
4 | jdMorgan | 7:30 pm June 9, 2006 | ||||
| Strange? Just a heads up |
4 | wilderness | 6:27 am June 8, 2006 | ||||
| mailer.hiphopcaucus.net Anyone see it? |
5 | youfoundjake | 6:21 am June 8, 2006 | ||||
| Surf Control ScSpider/0.2 |
5 | fiestagirl | 5:08 am June 8, 2006 | ||||
| GoogleBot/2.1 Has anyone else seen this UA? |
25 | GaryK | 1:36 am June 8, 2006 | ||||
| dp131.data.yahoo.com -- Mozilla/4.0 Yet Another Whatever from Yahoo that doesn't ask for robots.txt |
5 | Pfui | 12:06 am June 8, 2006 |
