Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt

Tell me about Robots.txt

         

sandeepsirohiseo

6:02 am on Jul 29, 2015 (gmt 0)

10+ Year Member Top Contributors Of The Month



Hello Friends,
I am new here, i have one query from all of you, When we use www.example.com/robots.txt to block some Folder and files. But i am confuse about one thing that
User-agent: *
Allow: /
Disallow: /Admin
Disallow: /Admin/
Disallow: Admin
----------------------------------
I want to know different between Disallow: /Admin, Disallow: /Admin/, Disallow: Admin.
Are there is any different between them or is it same. Or we can use any one.
Please help me, i am so confuse about this.

[edited by: bill at 10:16 pm (utc) on Jul 29, 2015]
[edit reason] use example.com [/edit]

goodroi

11:11 am on Jul 30, 2015 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



You might want to visit [robotstxt.org...] to learn the basics and then post questions if you still don't understand something.

sandeepsirohiseo

4:40 am on Aug 5, 2015 (gmt 0)

10+ Year Member Top Contributors Of The Month



Dear i know about robots.txt but i am asking different between
Disallow: /Admin,
Disallow: /Admin/,
Disallow: Admin.

not2easy

5:21 am on Aug 5, 2015 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Google's specifications can be studied here: [developers.google.com...]

The link supplied above describes the way most other robots read the robots.txt file.

lucy24

7:08 am on Aug 5, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



/admin is both a file and a directory, depending on what comes next.
/admin/ is a directory.
admin is a filename, but probably has no meaning in the context of robots.txt (because it doesn't begin with a / slash)

:: preparing for acute panic at discovery that robotstxt.org is apparently down for the moment ::

sandeepsirohiseo

6:58 am on Aug 6, 2015 (gmt 0)

10+ Year Member Top Contributors Of The Month



Thanks for your detail @Luck24 and @not2easy
Dear Luck24,
You mean if I use /Admin/ it will disallow only Admin directory,
And if i used /Admin then it will disallow admin directory and all files that comes after admin.

lucy24

6:14 pm on Aug 6, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



/admin/ means the /admin/ directory and all its contents.
/admin means the same thing BUT ALSO any top-level files called /admin.cgi or /administrator.html or /adminify.php or etcetera.
If your names don't happen to work that way, the final / makes no difference. The "vanilla" version of robots.txt-- the directives recognized by everyone, not just google-- are based on the beginning of the path: "match up to this point".