Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Add Sitemap To robots.txt For Autodiscovery


StupidScript - 9:17 pm on Apr 16, 2007 (gmt 0)


Thanks for your great script 'stupid script'.

I tried to mod it to index a php based forum like this without success.

I got it to generate OK but it didn't want to pick up the threads.

Any ideas how to fix it :)

Thanks, netchicken!

Are independent files stored somewhere for spiders to access? If so, run the script in that directory (modify $my_domain to match) then merge the generated file with the maps from other directories to create one master sitemap.

If there are no independent files for spiders to find, then the script will not be as useful to you.

NOTE: If the URLs that you want to include in one of these sitemaps look like the ones on this forum (with query strings) BE SURE to convert all of the entities in your URLs. i.e. If this is the type of URL you want in your XML sitemap: http:/ /www.example.com/pages.cgi?page=1&section=14 then be sure to convert it to: http:/ /www.example.com/pages.cgi?page=1[b]&[/b]section=14 See these instructions [sitemaps.org] for all entities that need to be converted to UTF-8 before including them in sitemap.xml.

[edited by: StupidScript at 9:18 pm (utc) on April 16, 2007]


Thread source:: http://www.webmasterworld.com/robots_txt/3310622.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com