Writting a search engine spider

Any ideas?

[chris_f]

There is another thread running similar to this at the moment but I am hoping for a few specific resources.

I am looking to create a spider (complete with customer UserAgent). Are there any resources out there that can teach people how to right a spider. I will be using ASP.net and VB.net.

Here are the stages which I will need to know.

1. How to bring back the source code from a url and place it in a string.
2. How to remove anything between the < and > (brackets inclusive).
3. How to place the contents of an unknown quantity of href tags into variables
4. How to send a custom UserAgent when getting the page.

Thanks in advanced.
Chris.

[NFFC]

We have just done one in VB, I'll get my man to post but it will be next week now.

[chris_f]

cool thanks.

Can you please sticky me you post it.

Thanks
Chris