Forum Moderators: goodroi
Actually i want to stop googlebot on all of my robots.txt pages but other SE can access only /my-data/ form but not others.
User-agent: googlebot
Disallow: /my-data/
User-agent: *
Disallow: /catalog/login.php
Disallow: /catalog/buy_it_now.php
Disallow: /catalog/add_product.php
Disallow: /login.php
Disallow: /buy_it_now.php
Disallow: /add_product.php
that will work or not?
thanks
One point to note is that you need a blank line just before the
User-agent: *
What you have there is telling Googlebot not to request anything starting with /my-data/
And you are telling all other bots not to request those others.
I think possible what you want is
User-agent: Googlebot
Disallow: /my-data/
Disallow: /catalog/login.php
Disallow: /catalog/buy_it_now.php
Disallow: /catalog/add_product.php
Disallow: /login.php
Disallow: /buy_it_now.php
Disallow: /add_product.php
User-agent: *
Disallow: /catalog/login.php
Disallow: /catalog/buy_it_now.php
Disallow: /catalog/add_product.php
Disallow: /login.php
Disallow: /buy_it_now.php
Disallow: /add_product.php
