Coding My Own Robot

spiders robots code

[tomandlis1]

Hi all,

I couldn't find a match for this topic, but since I figure the "cloaking" forum probably draws people who understand how to write code I thought I would post here.

I am trying to write my own spider for indexing internal content on my client's many sites. I am going to write it in .Net and have a backend DB of Oracle. Anybody have any idea where there is some good code I could use or at least some design guidance?

Also, if you want to talk me out of writing my own please feel free to post here too.

Thanks.

[johnhamman]

If you have a subscription to ASPtoday.com there is a good article on 'DATA MINING' which is pretty much a spider and on CsharpToday.com there is a spider article on building a spider. Its prettymuch screen scraping.
john

[tomandlis1]

Thanks I'll check it out. Do you know any open source bots?

[tomandlis1]

Did you mean the [asptoday.com...] article?

[johnhamman]

[asptoday.com...]
and
[asptoday.com...]
but im not sure there in .Net but it wouldnt take much to make it that way since it asp.

and on Csharptoday there is
A C# Web Spider written as a Web Service
[csharptoday.com...]

Building a special purpose search engine
[csharptoday.com...]

Both pretty good articles.
If you dont have a csharptoday membership email me for more details.
john

[johnhamman]

here is a new article from asptoday.com [asptoday.com...]