Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

IPs or domains crawled

Doubts as to whether spiders crawl IPs or domains



8:11 pm on Jul 30, 2002 (gmt 0)

Inactive Member
Account Expired


When some traditionally popular search engines (Lycos and Hotbot, for example) went from accepting URL submissions for free to charging for submission, it appeared that completely new domains (bought, registered and published after the switch to a 100% pay for inclusion method) HAD to pay to be included in those search engines while the sites that were already being spidered by these engines remained in their list of results.

However, I am unsure what exactly a spider remembers to crawl - a domain name or its actual IP server location. Basically, my problem is this - my site has existed for many years at a given domain and that domain does come up when relevant searches are conducted in Lycos, Hotbot, etc. However, after a recent redesign, I switched the hosting and moved the site to a new server. Will Lycos, Hotbot, and other search engines continue to crawl it as they do all the other domains that were in their list before they began to crawl or will it be unable to find the site, thus forcing me to pay for submission?

9:56 pm on July 30, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0


Do you have an ODP listing? How about Yahoo! ? Get them if you can... Usually a free ride into
almost all te top search engines as long as you don't mind waiting 3-4 months.

Search engines spider links. Once they have spidered your domain, they may record your IP address
and use it for most subsequent spidering, just to save DNS lookup time. But they do update stale
IP addresses periodically. So the answer to your question would be "both".

Any search engine that ignores the vast "unpaid" web is going to die if it tries to be a consumer
search engine (portal site). Overture survives because it sells paid listings to other portals
which also include unpaid lisitngs. As a web user, I would never use Overture search directly,
because by definition, only commercial sites are in its SERPs! - When I search, want to see the
widgets_dot_com_sucks.org site as well as the widgets.com site in the results!