Hi all, My goal is to learn how robots, crawlers and spiders deal with content on pages that have a redirect or meta refresh. If you have any advice about this and then how its affected with and without exceptions in the robots.txt file i would greatly appreciate!
As a noob working with php im guessing the ways of redirecting from one page to another are doing
- a simple meta-refresh
- a php header change
- some kind of javascript redirect
when a crawler comes across these kinds of page with redirects included does it -
a) scan the page and then jump to the redirected page
*result - both original and redirected page are indexed
b) scan the page and then stop
*result - only original page gets indexed
c) ignore original page, jump to redirect page and scan
* result - only redirect page gets indexed
I am curious as im using redirects a bit this week and wondering how relevant content(for indexing) would be treated on these 'original' pre-redirected pages.
thanks again
