Forum Moderators: open

Message Too Old, No Replies

Delay between Crawling and Indexing

How long it takes to be in index after crawling

         

sdani

6:45 pm on Feb 23, 2004 (gmt 0)

10+ Year Member



Yahoo's crawler took away about 20 pages from my site 10 days ago. They are not in the index yet. Does anyone knows how long it takes for those pages to show up in the index?
Google seems to be very fast at it. Whenever Google Crawls the pages, they show up withing couple of days.

sdani

6:52 pm on Feb 25, 2004 (gmt 0)

10+ Year Member



bump

allanp73

9:31 pm on Feb 25, 2004 (gmt 0)

10+ Year Member



I noticed the same thing. It crawled one of my site about a month ago and is just getting into the Yahoo surps now.
Do you know where the Yahoo crawler free submit is?

farside847

10:01 pm on Feb 25, 2004 (gmt 0)

10+ Year Member



>Does anyone knows how long it takes for those pages to show up in the index?

Not yet, the crawler is still too new to tell. Most likely 3 to 5 weeks, but
that is based on absolutely no hard evidence. ;)

>Do you know where the Yahoo crawler free submit is?

You can suggest your site to be listed in the Yahoo Directory at
[docs.yahoo.com...]

Once your site is listed in the directory it should be crawled. But, there is
no submit to just get crawled.

allanp73

10:07 pm on Feb 25, 2004 (gmt 0)

10+ Year Member



farside847,

This is for Yahoo directory. Isn't there anything which submits to Yahoo surp or that is free?

farside847

10:34 pm on Feb 25, 2004 (gmt 0)

10+ Year Member



All of my sites (content and retail) listed in the Y! directory are being crawled daily by Slurp. Of course, it could be coincidence, but I doubt it. The Y! bot needs some place to start its crawl from, why not its own directory? Similarly, Google uses DMOZ this way, to a point...

steveb

11:15 pm on Feb 25, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



allanp73, try...

[submitit.bcentral.com...]

But otherwise I think the answer is "No" as of today.

allanp73

11:26 pm on Feb 25, 2004 (gmt 0)

10+ Year Member



Excellent, thanks Steveb

GreenLeaf

1:52 pm on Feb 26, 2004 (gmt 0)

10+ Year Member



My site IS in Yahoo directory, but Yahoo! Slurp comes and grabs only robots.txt and my site isn't in Yahoo search.

Net_Wizard

3:38 am on Feb 27, 2004 (gmt 0)



In fairness to Yahoo :)...

Slurp went through one of my site a few days ago and crawled several hundred pages and are now in Yahoo serp.

Traffic so far is nothing really to be excited about, perhaps my pages are not just optimize for Yahoo.

Sort of a preview of what kind and how much traffic can I get from Yahoo if I have paid for those pages.

mayor

6:17 am on Feb 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Here's what I think I saw happening:

1. At first, the new search Yahoo displayed sites mostly from the Inktomi data base using the Yahoo query 'site:mysite.com'

2. Then Google (or those Google sites that Yahoo had been using) was rolled back in, but only as backfill. At that time, the Yahoo query 'site:mysite.com' began showing the Google index sites, but since the Google index pages are only backfill, they get traffic only for very unique and unusual searches, which is next to nothing.

3. So now, the presence of your pages in the oganic ('free') active search Yahoo data base can only be identified by actual search.yahoo traffic volume from log analysis. Otherwise, the best available indicator of pages that will draw search.yahoo traffic are pages indexed by Inktomi.

Can anyone confirm my theory, which is based on my limited observations?

2_much

1:57 am on Mar 1, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Reps from Yahoo were saying at the WebmasterWorld conference that they will be adding a free "add url" link soon so people can submit pages to be indexed.

They also said they will be adding sites from the directory to their search database. However, I got the impression that those sites will be re-reviewed (by either bots or people) before they get added, so the spammy sites won't make it in.