Forum Moderators: DixonJones
Q7 states;
I haven't got access to my log file, how do I see if Googlebot spiders my pages.
Use a server side include (eg. Apache XSSI, PHP or ASP) to embed or call a script that checks for:
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)" as the USER_AGENT or "crawl*.googlebot.com" or "crawler*.googlebot.com" as the HOST. You cannot use an image or Javascript based tracker because Googlebot won't trigger it.
Does anyone have the relevant SSInclude for an .asp page that I can just copy and paste onto a page? Also, could this bit of code be extended to include other bot info?