Forum Moderators: goodroi
I have included in my robots.txt the follow:
Disallow: /?
Disallow: */?
Disallow: */?*
This has been added because I wanted to prevent any spiders from crawling the dynamic pages.
My question is if a dynamic page is linked elsewhere on the web (a completely different site) and the spider crawls that site then see the link...will the spider crawl that link.
====================================
What am I trying to accomplish?
I have a blog hosting website. Currently the URL's are not search engine friendly they are dynamic (e.g. [example.com...]
I know spiders do not like dynamic pages that is why I disallowed them from being crawled. But I would like the author of the (dynamic) page to be able to advertise its blog/link. Will me disallowing the crawl from the site location affect the link from being crawled from another location. Like if advertised on another website will the spider see the link and grab it and index it. Or will the spider see the link, then talk to my robots.txt file, and spit the link out. Please share you knowledge. Any thoughts or suggestions is appreciated. Thanks.
[edited by: pageoneresults at 6:49 pm (utc) on Mar. 9, 2005]
[edit reason] Examplified URI Reference [/edit]
