Forum Moderators: goodroi
Im working on my new domain, still not setup directories yet so i wasnt think to put my robots.txt since ask (jeeves) bot find my site after 2 days i bought the domain .
As a newbie i need help pls, i have an includes folder which contains css folder js folder and footer, header,menu,content files in it for SSI.My question is can i dsallow all includes folder? (Disallow: /includes/ ) or bots cannot crawl my pages or is there any way/suggestion.
One more stupid question but i need to make it clear, do i need to disallow .htaccess or .stacess? if yes can you give me example path please .
Thanks
As SSI actually get served up as part of the contents of a page, excluding the /includes/ folder will have no effect, as the spider/bot won't know that where that contents is coming from.
Open the page in a browser, and view the source, that is what the bot will see.
If it is just menu's, header's and footer's I wouldn't worry, as long as there is enough other contents on the page.
