Changes between Version 4 and Version 5 of DeveloperPage


Ignore:
Timestamp:
Feb 6, 2007, 7:07:27 PM (17 years ago)
Author:
Jari Häkkinen
Comment:

robots.txt information added.

Legend:

Unmodified
Added
Removed
Modified

  • DeveloperPage

    v4 v5  
    4949
    5050Branches can be treated correspondingly.
     51
     52
     53== robots.txt ==
     54
     55The web page indexing robots are killing the web server since the robots index '''every''' revision of the source. There is a RobotsTxt file with information to well behaved robots. The contents of RobotsTxt is the same traditional robots.txt, and the [http://trac-hacks.org/wiki/RobotsTxtPlugin RobotsTxtPlugin] plug-in will serve a http://base.thep.lu.se/robots.txt to the robots.