[pmwiki-users] Linkchecker and robots.txt
chr at home.se
chr at home.se
Thu Dec 15 10:42:20 CST 2005
Hi
I'm testing the W3C linkchecker:
http://validator.w3.org/checklink
which can also be installed and used locally.
However, in order to use it a small modification is needed in robots.txt,
or it'll be disallowed otherwise. Here's what you need to append:
User-Agent: W3C-checklink
Disallow:
Would this make sense as a default? (In case robots.txt comes with
pmwiki... I can't remember)
/Christian
--
Christian Ridderström, +46-8-768 39 44 http://www.md.kth.se/~chr
More information about the pmwiki-users
mailing list