Welcome to WebmasterWorld Guest from 23.22.140.143

Forum Moderators: open

Message Too Old, No Replies

Inktomi crawling dynamic pages?

     
1:33 am on Jul 11, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Has anyone seen Inktomi crawling dynamic pages lately? It's been hitting lots of mine in the last couple of weeks, but none of them have made it into the index.

The Slurp info page says:

Slurp now has the ability to crawl dynamic links or dynamically generated documents. It will not, however, crawl them by default.

I'm not sure what that means. Maybe it crawls them for links but doesn't actually list them?

6:13 am on July 12, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2001
posts:2059
votes: 0


Yes inktomi have been crawling some of my dynamic pages for quite a while, to me it seems to be a bit harder to get dynamic url's indexed for free if you compare with Google and Fast.
5:56 pm on July 12, 2003 (gmt 0)

New User

10+ Year Member

joined:Mar 31, 2003
posts:13
votes: 0


Very recently Inktomi crawled my dynamic pages and has so far indexed around 300 of them. (I've not paid for any inclusion.) Google, on the other hand, has yet to even crawl beyond the three static pages on the same site, the index page only of which has been in the G. index for about one year.
4:18 pm on July 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 18, 2001
posts:1042
votes: 0


I am not sure what triggers Ink to crawl deep into dynamic pages (lots of inbound links?). All I can say is that once Slurp decides to do it they will go deep, over time. As verbum states, deeper than Google in many instances.
10:36 am on July 14, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7242
votes: 0


I can vouch for that Brad - have had Inktomi going very hard at our site for the last week. Very deep and grabbing most of the dynamic URL's (some with three or four variables).

We have not paid for inclusion.

TJ

7:03 am on July 15, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:June 6, 2003
posts:162
votes: 0


I've seen Inktomi crawling my dynamic pages for quite awhile. However, I just assumed that they wouldn't index them. I got that impression from the blurb at [inktomi.com...] which states "dynamically generated links and documents can be different for every retrieval so there is no use in indexing them.". However, I didn't really understand the reasoning. The dynamically generated pages I usually see seem to remain just as stable as static HTML pages for any given URL.
3:09 am on Aug 13, 2003 (gmt 0)

Full Member

10+ Year Member

joined:May 14, 2003
posts:215
votes: 0


i donno if inktomi is on drugs
but it hasnt been spidering my dynamic urls properly
for some reason it decides to look for
domain.com/subdirectory
instead of
domain.com/subdirectory/

every other spider does the second, but because inktomi does the first it gets a 404.

Is there a reason it does that?
google, ask have done it properly , as well as wisenutbot.

anyone have a clue?

4:32 am on Aug 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Yeah, I noticed that. It always strips the last slash. I don't know why, except that Slurp is pretty buggy. Any chance of fixing your pages to either work without the slash, or return a 301 redirect?
2:29 pm on Aug 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:960
votes: 0


They have opened a can of worms ;)

A search for aluminum element [search.positiontech.com] returns me lovely spammy results :)

On the second page, there are several those-long-titled-results which redirect to one commercial site..

Is it so easy to spam ink?

3:10 pm on Aug 13, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Oh, God.