Changes between Version 9 and Version 10 of DeveloperPage


Ignore:
Timestamp:
May 29, 2007, 12:09:27 PM (17 years ago)
Author:
Jari Häkkinen
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • DeveloperPage

    v9 v10  
    5959== robots.txt ==
    6060
    61 The web page indexing robots are killing the web server since the robots index '''every''' revision of the source. There is a RobotsTxt file with information to well behaved robots. The contents of RobotsTxt is the same traditional robots.txt, and the [http://trac-hacks.org/wiki/RobotsTxtPlugin RobotsTxtPlugin] plug-in will serve a http://base.thep.lu.se/robots.txt to the robots.
     61The web page indexing robots are killing the web server since the robots index '''every''' revision of the source. There is a RobotsTxt file with information to well behaved robots. The contents of RobotsTxt is the same traditional robots.txt, and the [http://trac-hacks.org/wiki/RobotsTxtPlugin RobotsTxtPlugin] plug-in will serve a http://baseplugins.thep.lu.se/robots.txt to the robots.