What bots ignore robots.txt?

[Jon12345]

Ok, so there are various bots and crawlers parading around the internet. I am using this...

User-agent: *
Disallow: red.php

...in my robots.txt file. I presume this is correct if I don't want any bots to go to follow to the red.php page. yes?

But what percentage of bots actually ignore such a request? Any idea?

Also, should I use a no-follow tag instead?

Thanks,

Jon

[Dijkgraaf]

Not quite correct, it should have a leading slash before the page name as below.

User-agent: *
Disallow: /red.php

This will tell bots not to do a GET /red.php from your domain.

Any bots that don't obey robots.txt are unlikely to obey the nofollow tags.

[Jon12345]

Are you saying my version won't work at all?

[Lord Majestic]

Are you saying my version won't work at all?

Yes, your version is incorrect because robots.txt does not specify pattern or sub-string match, but requires to exclude URLs that start with provided value. Since all URLs start with /, it is essential to include said /.

[Jon12345]

Ok, thank you. I have changed it now.

[Jioffri]

Jon,

Would the following also work to block the red.php?

User-agent: red.php
Disallow: /

Or is it more correct to use the example you gave to deny every bot a person wants to block and list them individually beneath in a series on separate lines as the need arises?

[Dijkgraaf]

Jioffri, your example is saying that a BOT called red.php isn't allowed anything starting with / (i.e. the whole site).

[Jioffri]

Ok, thanks! I understand now.