Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Do spiders crawl popups & drop down menus?



6:27 pm on Jul 16, 2008 (gmt 0)

5+ Year Member

Hi guys :)

I was under the impression that spiders can't crawl JavaScript popups and drop down menus.

But when I recently looked in WMT, we have Duplicate Title Tag issues from popup and drop down URLs.

For example, on our site, you can choose how many results to view (10, 20, 30, etc.) via a drop-down menu. The URLs of these pages vary slightly but all have the same title tag. The only way of accessing the links is through the drop down menu.

Should I <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> these pages to get rid of the duplicate title tag issue?

Any insight is really appreciated.

Thanks a ton!

Receptional Andy

6:32 pm on Jul 16, 2008 (gmt 0)

Generally speaking, spiders can extract URLs and similar from javascript, which means that links in drop-down menus and similar can be spidered and indexed.

In addition, Googlebot has somewhat 'experimental' crawling of HTML forms [webmasterworld.com].

So, realistically, you have to consider content accessible via forms (javascript-based or not) to be spiderable, and if you don't want such content indexed, robots exclusion is the way to go. You could do this via a robots.txt file (typically utilising the major search engines' support for wildcard entries) or a meta tag is fine. If you think there are links from the pages that are worth spidering, you might consider noindex,follow.


1:22 pm on Jul 17, 2008 (gmt 0)

5+ Year Member

Perfect, thank you so much!

Featured Threads

Hot Threads This Week

Hot Threads This Month