homepage Welcome to WebmasterWorld Guest from 174.129.80.166
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Visit PubCon.com
Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
Forum Library, Charter, Moderator: open

Deprecated - Search Engine Submission Forum

    
IPs or domains crawled
Doubts as to whether spiders crawl IPs or domains
cbc77




msg:703260
 8:11 pm on Jul 30, 2002 (gmt 0)

When some traditionally popular search engines (Lycos and Hotbot, for example) went from accepting URL submissions for free to charging for submission, it appeared that completely new domains (bought, registered and published after the switch to a 100% pay for inclusion method) HAD to pay to be included in those search engines while the sites that were already being spidered by these engines remained in their list of results.

However, I am unsure what exactly a spider remembers to crawl - a domain name or its actual IP server location. Basically, my problem is this - my site has existed for many years at a given domain and that domain does come up when relevant searches are conducted in Lycos, Hotbot, etc. However, after a recent redesign, I switched the hosting and moved the site to a new server. Will Lycos, Hotbot, and other search engines continue to crawl it as they do all the other domains that were in their list before they began to crawl or will it be unable to find the site, thus forcing me to pay for submission?

 

jdMorgan




msg:703261
 9:56 pm on Jul 30, 2002 (gmt 0)

cbc77,

Do you have an ODP listing? How about Yahoo! ? Get them if you can... Usually a free ride into
almost all te top search engines as long as you don't mind waiting 3-4 months.

Search engines spider links. Once they have spidered your domain, they may record your IP address
and use it for most subsequent spidering, just to save DNS lookup time. But they do update stale
IP addresses periodically. So the answer to your question would be "both".

Any search engine that ignores the vast "unpaid" web is going to die if it tries to be a consumer
search engine (portal site). Overture survives because it sells paid listings to other portals
which also include unpaid lisitngs. As a web user, I would never use Overture search directly,
because by definition, only commercial sites are in its SERPs! - When I search, want to see the
widgets_dot_com_sucks.org site as well as the widgets.com site in the results!

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved