homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
Forum Library, Charter, Moderator: open

Deprecated - Search Engine Submission Forum

IPs or domains crawled
Doubts as to whether spiders crawl IPs or domains

10+ Year Member

Msg#: 406 posted 8:11 pm on Jul 30, 2002 (gmt 0)

When some traditionally popular search engines (Lycos and Hotbot, for example) went from accepting URL submissions for free to charging for submission, it appeared that completely new domains (bought, registered and published after the switch to a 100% pay for inclusion method) HAD to pay to be included in those search engines while the sites that were already being spidered by these engines remained in their list of results.

However, I am unsure what exactly a spider remembers to crawl - a domain name or its actual IP server location. Basically, my problem is this - my site has existed for many years at a given domain and that domain does come up when relevant searches are conducted in Lycos, Hotbot, etc. However, after a recent redesign, I switched the hosting and moved the site to a new server. Will Lycos, Hotbot, and other search engines continue to crawl it as they do all the other domains that were in their list before they began to crawl or will it be unable to find the site, thus forcing me to pay for submission?



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 406 posted 9:56 pm on Jul 30, 2002 (gmt 0)


Do you have an ODP listing? How about Yahoo! ? Get them if you can... Usually a free ride into
almost all te top search engines as long as you don't mind waiting 3-4 months.

Search engines spider links. Once they have spidered your domain, they may record your IP address
and use it for most subsequent spidering, just to save DNS lookup time. But they do update stale
IP addresses periodically. So the answer to your question would be "both".

Any search engine that ignores the vast "unpaid" web is going to die if it tries to be a consumer
search engine (portal site). Overture survives because it sells paid listings to other portals
which also include unpaid lisitngs. As a web user, I would never use Overture search directly,
because by definition, only commercial sites are in its SERPs! - When I search, want to see the
widgets_dot_com_sucks.org site as well as the widgets.com site in the results!


Global Options:
 top home search open messages active posts  

Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved