Welcome to WebmasterWorld Guest from 54.161.209.240

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Do spiders crawl popups & drop down menus?

     
6:27 pm on Jul 16, 2008 (gmt 0)

New User

5+ Year Member

joined:Sept 14, 2007
posts:12
votes: 0


Hi guys :)

I was under the impression that spiders can't crawl JavaScript popups and drop down menus.

But when I recently looked in WMT, we have Duplicate Title Tag issues from popup and drop down URLs.

For example, on our site, you can choose how many results to view (10, 20, 30, etc.) via a drop-down menu. The URLs of these pages vary slightly but all have the same title tag. The only way of accessing the links is through the drop down menu.

Should I <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> these pages to get rid of the duplicate title tag issue?

Any insight is really appreciated.

Thanks a ton!

6:32 pm on July 16, 2008 (gmt 0)

Senior Member

joined:Jan 27, 2003
posts:2534
votes: 0


Generally speaking, spiders can extract URLs and similar from javascript, which means that links in drop-down menus and similar can be spidered and indexed.

In addition, Googlebot has somewhat 'experimental' crawling of HTML forms [webmasterworld.com].

So, realistically, you have to consider content accessible via forms (javascript-based or not) to be spiderable, and if you don't want such content indexed, robots exclusion is the way to go. You could do this via a robots.txt file (typically utilising the major search engines' support for wildcard entries) or a meta tag is fine. If you think there are links from the pages that are worth spidering, you might consider noindex,follow.

1:22 pm on July 17, 2008 (gmt 0)

New User

5+ Year Member

joined:Sept 14, 2007
posts:12
votes: 0


Perfect, thank you so much!