Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

Can you see what pages Slurp visits?

(within your site)



8:15 pm on Aug 21, 2006 (gmt 0)

10+ Year Member

I can see which robots have visited our site, but can I tell if a robot has visited a specific page within my site?

We don't have any robot.txt tags on our pages, so there should be nothing telling Yahoo not to index our pages.

Our issue is that some pages have been indexed and some are not found at all (but are found on Google and MSN and ranking well).


11:53 pm on Aug 21, 2006 (gmt 0)

10+ Year Member

Yeah, find your apache logs and every line should show what page was hit along with what time, etc....


4:01 am on Aug 22, 2006 (gmt 0)

10+ Year Member

That is something I considered, but I was hoping to avoid going through the log files (this is a very large site), and I will have to request those files because I only have access to our CMS.

Thank you for the tip.


8:52 pm on Aug 26, 2006 (gmt 0)

10+ Year Member

There is no way around it,if you dont have password protected stats you can access, then request the logs.
If you don't have a program for reading the logs just rename access_log to .txt and use wordpad. Use the edit-find on this page command and enter ysearch/slurp for a quick scan.

Featured Threads

Hot Threads This Week

Hot Threads This Month