Rapid Inclusion in Search Engines

[farid]

I'm doing some research for college work on search engines

I have found reading about search engines very interesting.

What i would like to find out is, when a website is submitted to a search engine on some it can take many months before it is included. But then there are companies that can include a website in search engines in a matter of days. How is this possible? I know some charge fees for this rapid inclusion service, but do they twick the coding, i.e keyword tags or use special software?

look forward to your comments

[troels nybo nielsen]

Welcome to WebmasterWorld, farid.

> in a matter of days.

At least one search engine (Gigablast) may sometimes spider and index a page within minutes after submitting.

> How is this possible?

Simply a question of putting you forward in the queue.

> do they twick the coding, i.e keyword tags

AFAIK they don't.

[Web Footed Newbie]

Welcome to WW farid,

But then there are companies that can include a website in search engines in a matter of days. How is this possible?

Different search engines (which are different companies) are like individuals, everyone is different. If you are researching, perhaps typing in "search engine optimization" in your search, you will learn quite a bit.

SEO is the practice of setting up a web page (a page, not a site) where the page is "friendly" (readable) to search engines. BOTS, or crawlers, visit sites, looking for certain, standard information in the HTML code, like the TITLE and DESCRIPTION. It is the crawlers timing, and the importance of the page title, supported by the page content, that helps make the page friendly to search engines, and for the page to rank well. Some crawlers visit quit quickly, others visit on their own schedule.

There are some search engines you can pay for inclusion (PFI), which will visit you very quickly.

There's my two cents. Read more here at WW, and you will be able to not only write a report, but perhaps several books on the subject of search engines!
Welcome again,
WFN

[simonuk]

The fastest way to get indexed is by having links going to your site from a site that is updated often.

I have three sites I update daily and whenever I upload a new site I link on these 3 sites.

The quickest I've had a site indexed is 30 minutes and the average is 1 to 3 days.

Because the 3 sites are updated so often google crawls them often.

Simon.

[zwingo]

Simon,

How do you get the SE's to update your site that often?

[simonuk]

It's because the content is updated 3 or 4 times a day every day on both the main page and in my blog which is also on the site. Google loves sites that update very often and will crawl them much more often than a site that never gets updated. I've seen it crawl and update my main site more than once in one day before.

For me it's worth the effort because every new site I put online for a client gets added same day or very soon after. Giving them a nice big free PR boost also keeps them happy :)

By me having 3 sites I update daily I have a very good chance that google will crawl one well enough to add the site super fast.

I haven't bothered submitting to search engines for a long time now, never need to anymore.

Hope that helps.

Simon.

[Navdeep]

hi simon,
have you submitted these three sites(that you update daily), to the search engine when you started optimizing your main site.

[simonuk]

I submitted my main site once about 5 years ago. Other than that one time that's it. Never bother submitting anything anymore :)

Simon.

[zwingo]

Thanks Simon...I have a similar site to the three you have. I updated it once or twice a month and include links on new sites for the purpose of getting spidered faster than via manual submission.

I was not aware a site gets spidered more often by Google if it is updated more often.

Do you see any other SE's that work this way?

Thanks again for the tip.

Jerry

[simonuk]

To be honest Jerry I only bother checking against google because it feeds so many other search engines as well.

Simon.

[troels nybo nielsen]

Welcome to WebmasterWorld, Simon. I look forward to more posts from your hand.

I agree in everything you have written and only have one thing to add. There is one more factor that most likely has a rather large influence on the willingness of search engines to spider a website often: Its "importance" which in Google terms is largely identical with its PageRank. I am fairly sure that I can see that factor in the different spidering patterns on my own websites.

Do you see anything like this?

Troels

[Navdeep]

hi all,
i am new in this field, i have a doubt.
lets say i have a site with home page "query.html" and i submitt a home page to the SE, then i create another page with different domain name say www.query1.com/index1.html and different meta keywords/description, then i submitt this page to the SE, but i the coding i redirect this page to my original page query.html with no other contents in index1.html. when user click the URL of index1.html it will be redirected to query.html.
then i create another page with different domain name www.query2.com/index2.html again redirecting the user to main page query.html. these pages has no other purpose, just to redirect the user to query.html. i will submitt index2.html to the SE. now when ever user search these all will appear in search engine listing and user clicking any one of the URL will go to my main page query.html. thus the chance of getting hits is increased.
now question is that, is this method is worth doing to increase traffic?
please give your expert comments on this

[simonuk]

hello troels nybo nielsen, thanks for the welcome :)

I haven't seen the PR playing too much into it. My main site wasn't updated a few times a day until recently (6 months ago) when I started blogging and adding more daily wallpapers to the site. It had a PR of only 2 but google was indexing it daily'ish regardless. It's now on PR5 (soon to be 6 hopefully) and I haven't noticed any difference in google crawling the site. It just started crawling often when I started updating often. Google took a few weeks to figure out I had turned a semi-static site into a daily updated site but once it knew that there was no stopping it coming back for more :)

My blog is set is archive each day which generates a bucket load of new pages and I manually update the date on the frontpage each time I update the blog so google knows I've been busy. It still takes time for it to perform a deep crawl (think it only does them once a month) but the frontpage is always being re-cached.

I'm not 100% sure about the next bit but I'll share anyway to see what others have found...

I've been a web designer since '96 and some of my orignal sites are still online to this day. What I've started doing is link exchanging on them and I have noticed that google is caching them more often than before. Again, I update the links every day. It doesn't crawl the pages as often as my desktop wallpaper site but then I'm not updating the main pages, only the links pages.

Google does like activity on a site and it certainly speeds up the process for adding new sites from what I've found.

Simon.

<edited spelling>