How do bots and humans find brand new sites?

[BlueSky]

I purchased a new domain two days ago. My logs already show I've been visited by three bots and what appears to be some regular surfers. They're being served 401's right now because I'm not anywhere near ready to bring the site live.

The domain was previously registered but hasn't been used in a very long time. It doesn't appear in any search engine and wayback shows nothing since 1999. My other domain was never used at all yet the same thing happened.

How are these guys finding out I registered a new domain and so fast? Is there a service or site which tracks new registrations and gives this info out?

[sonik]

I know there are services that watch for expired domains like a hawk, so I imagine it goes the other way around also. I'm sure spammers would want to send out "info" to [e-mail] and whatever other addresses are listed in the whois info.

[John_Caius]

Are there any links to the domain? That's how domains get found by spiders. Type link:www.yourdomainname.com into www.alltheweb.com to see whether there are any.

[BlueSky]

No links at all. That's why I'm so puzzled how they found it. This particular site is gong to be commercial. I haven't even registered the business name with my state yet.

Two of the bots are Inktomi and ia_archiver. The third one is an email collector. The others have no UA and I'm not letting them in to find out their real intention. It's just really weird how they found it so fast. Maybe like you said there's some sort of service that monitors the whois database.

[cfx211]

I have wondered about this too. Do you think hosting companies and/or domain registrars generate a page full of newly added domain links or maybe have an alpha listing site map out there? Could be these pages are setup for admin purposes but are getting found by spiders.

[pmkpmk]

Check out www . whois . sc - registered surfers can automatically monitor keywords for domains.

It's pretty useful. If you have e.g. a trademark for "blue widgets", you can enter these keywords into the monitor list and get notice whenever a new domain containing "blue widget" comes to life.

[edit]Oh... registration is free btw.[/edit]

[edited by: pmkpmk at 5:13 pm (utc) on Aug. 20, 2003]

[jimbeetle]

The domain was previously registered

That's the key in this instance. The domain name is still in some databases or has some old links from other sites.

Try a link.all:www.domain.tld -site:www.domain.tld on alltheweb to see if any old links pop up.