QNetwork - 4:11 pm on Oct 3, 2002 (gmt 0) Can you guys suggest anything about this behavior?
After a month, deep crawler came to my site last night. Just grabbed 2 pages and went away. Came back again this morning and grabbed another 2 pages. It is hitting my login page (when trying to access the secured pages) and going away from that. I do not want the bot to crawl my secure pages. From the index page, it just went to one of the inside link and got the login page.
Can you guys suggest anything about this behavior?