So I was assigned a project to research for information different web crawlers that can be used. I'm supposed to compile a list with their different features, designs, specifications, etc (I don't even know what exactly I would look for in detail about Web Crawlers). Could anyone provide some information or maybe a list of specific Web Crawlers that I could research to get me going?
Thanks!