Retrieve and store temperature and humidity measurements

[Nicklas Nordborg]

New 'climate' sensors have been installed in the lab and they are connected to the network! It would be nice if Reggie could query the sensors at regular intervals and store temperature and humidity on the server.

[olle]

Ticket accepted.

[olle]

Background information:

• A number of climate sensors have been purchased and installed at the department of Oncology at Medicon Village/Lund University. Four sensors have been purchased, but only no. 1,2, and 4 were working at the end of February, 2014. On 2014-03-31 finally sensor no. 3 was on-line, but sensor no. 1 had then been off-line for unknown reasons for some days.
• The sensors are of type Thermo Recoder TR-702W from ThermoWorks. For product information, check ​http://www.thermoworks.com/products/logger/t-and-d-tr700series.html. They can be configured to be accessed over the network from a web browser, with a page title indicating the lab/room a sensor is placed in.
• The web page shown, when a sensor is accessed over the network with the standard URL, displays current as well as older values of temperature and humidity (in units of °C and %, respectively). Current values are displayed as numbers, while a graph shows older values (the numerical values for a specific time can be found by placing the mouse pointer over the graph at the point of interest). The web page is customizable to a large degree by a number of buttons, and the graph can be zoomed by dragging the mouse over a time region. One button displays current settings.
• With all flexibility of the current sensor web page, there are still some limitations. One is that the graph cannot display more than about 12 hours of measurements, making it hard to check changes in the lab environment over longer time periods.
• A specification man_format-xml-122-all-eng.pdf of the data format used by the sensors for sending data can be downloaded from ​http://www.tandd.com/support/download/manual/xml_format.html (no direct link available).

Data for communicating with the sensors are as follows (2014-04-02):

URL	Title	Serial Number	IP-number	Comment
​http://givare1.onk.lu.se	G1 Maskinrummet (B32C2)	48740059	130.235.5.197	On-line
​http://givare2.onk.lu.se	G2 RNAlab (B33A2)	4874007D	130.235.5.198	On-line
​http://givare3.onk.lu.se	G3 Apparatrummet (C21B3)	48740075	130.235.5.199	On-line
​http://givare4.onk.lu.se	G4 Lilla apparatrummet (C22A3)	4874007B	130.235.5.200	On-line

[olle]

Retrieval of sensor data:

Inspection of the sensor web page code and the data format specification in document man_format-xml-122-all-eng.pdf mentioned above, revealed the following (Nicklas Nordborg gave invaluable help here):

• Adding "/current.inc", "/params.inc", or "/datas.inc" to a sensor URL resulted in the return of different data in a JSON-style format.
• Adding "/current.inc" to a sensor URL returned JSON-style data, that after Base64-decoding and conversion among other things contained serial number of the sensor, current local time, temperature in °C, and humidity in %. The latter two values were given with an accuracy of one decimal.
• Adding "/params.inc" to a sensor URL returned JSON-style data, that after Base64-decoding and conversion among other things contained the title of the sensor, indicating the lab/room it was installed in.

To test that the principles stated above were fully understood, a "dummy" application was implemented consisting of a JSP script and a Java servlet. The HTML page with the JSP script contained a pop-up menu to select a sensor plus a button to fetch its data by sending an Ajax request to the servlet. Tests showed that it was possible to retrieve sensor title, serial number, current local time, temperature, and humidity values that were consistent with what was returned from the original sensor web page. All data retrieval and conversion in the dummy test application were performed by the Java servlet, without use of any JavaScript in the HTML page, unlike the original sensor web page.

[olle]

Detailed examples for retrieving sensor data from climate sensors:

Even though all steps are simple, the full procedure to retrieve a sensor value from a climate sensor can be hard to find from scratch, so some examples in Java are therefore described here.

General:

1. The raw data content for a url is obtained by a HTTP "get" request, e.g. performed using the Apache Commons HttpClient package. In the example below, the raw data content is contained in String variable "content" (output of error messages has been simplified):

   import org.apache.commons.httpclient.HttpClient;
   import org.apache.commons.httpclient.HttpStatus;
   import org.apache.commons.httpclient.methods.GetMethod;
   
   private String fetchDataFromUrl(String url)
   	throws ServletException, IOException 
   {
   	String content = null;
   	GetMethod getMethod = null;
   	try
   	{
   		getMethod = new GetMethod(url);
   		HttpClient httpClient = new HttpClient();
   		int statusCode = httpClient.executeMethod(getMethod);
   		// Check status code
   		if (statusCode != HttpStatus.SC_OK)
   		{
   			System.out.println("Error when trying to execute GetMethod: " + getMethod.getStatusLine());
   		}
   		content = getMethod.getResponseBodyAsString();
   	}
   	catch (IOException e)
   	{
   		System.out.println("IOException when trying to execute HttpClient method : " + e);
   	}
   	finally
   	{
   		// Release current connection to the connection pool once you are done
   		if (getMethod != null)
   		{
   			getMethod.releaseConnection();
   		}
   	}
   	return content;
   }
   

2. The raw data content is processed into a standard JSON-style string by removing odd characters before first brace, and replacing single quotes with double:

   	String jsonStr = "" + content;
   	// Remove odd character(s) before first left brace "{"
   	int leftBraceIndex = jsonStr.indexOf("{");
   	if (leftBraceIndex >= 0)
   	{
   		jsonStr = jsonStr.substring(leftBraceIndex);
   	}
   	jsonStr = jsonStr.replaceAll("\\'", "\\\"");
   

3. Retrieval of sensor title, exemplified by sensor URL http://givare2.onk.lu.se/:
   a. String content = fetchDataFromUrl("http://givare2.onk.lu.se/params.inc");
   b. A standard JSON-style string jsonStr is obtained by processing content according to step 2 above.
   c. The sensor title labSensorName is obtained using the open-source class Base64Coder (see ​http://www.source-code.biz for details):

   // Initialize lab sensor name
   String labSensorName = "";
   
   try
   {
   	JSONParser parser = new JSONParser();
   	JSONObject jsonOutput = (JSONObject) parser.parse(jsonStr);
   	String d30 = (String) jsonOutput.get("D30");
   	// Decode Base64-encoded data
   	byte[] d30ByteArray = Base64Coder.decode(d30.toCharArray());
   	// Get sensor name from beginning of Base64-decoded byte array
   	int arraySize = d30ByteArray.length;
   	for (int i = 0; i < arraySize; i++)
   	{
   		char curChar = (char) d30ByteArray[i];
   		if (curChar != (char) 0x00)
   		{
   			labSensorName += curChar;
   		}
   		else
   		{
   			// Stop after first null byte
   			break;
   		}
   	}
   }
   catch (ParseException e)
   {
   	System.out.println("JSON ParseException e = " + e);
   }
   

4. Retrieval of sensor serial number, local measurement time, temperature, and humidity values, exemplified by sensor URL http://givare2.onk.lu.se/:
   a. String content = fetchDataFromUrl("http://givare2.onk.lu.se/current.inc");
   b. A standard JSON-style string jsonStr is obtained by processing content according to step 2 above.
   c. The sensor serial number, local measurement time, temperature, and humidity values are obtained using the open-source class Base64Coder (see ​http://www.source-code.biz for details):

   // Initialize lab temperature and humidity data
   String serialNumber = "";
   String dateTimeStr = null;
   String temperatureStr = null;
   String humidityStr = null;
   
   try
   {
   	JSONParser parser = new JSONParser();
   	JSONObject jsonOutput = (JSONObject) parser.parse(jsonStr);
   	String d20 = (String) jsonOutput.get("D20");
   	String d21 = (String) jsonOutput.get("D21");
   	// Decode Base64-encoded data
   	byte[] d20ByteArray = Base64Coder.decode(d20.toCharArray());
   	byte[] d21ByteArray = Base64Coder.decode(d21.toCharArray());
   	// Get sensor serial number from Base64-decoded byte array
   	int arraySize = d21ByteArray.length;
   	for (int i = 0; i < 8 && (10 + i) < arraySize ; i++)
   	{
   		char curChar = (char) d21ByteArray[10 + i];
   		if (curChar != (char) 0x00)
   		{
   			serialNumber += curChar;
   		}
   	}
   	// Get raw variable values from Base64-decoded byte array
   	int ch1 = (d20ByteArray[26] & 0xFF) | ((d20ByteArray[27] & 0xFF) << 8);
   	int ch2 = (d20ByteArray[42] & 0xFF) | ((d20ByteArray[43] & 0xFF) << 8);
   	// Variable nowTime is the number of seconds since 1070-01-01 00:00:00
   	long nowTime = (d20ByteArray[6] & 0xFF) | ((d20ByteArray[7] & 0xFF) << 8) | ((d20ByteArray[8] & 0xFF) << 16) | ((d20ByteArray[9] & 0xFF) << 24);
   	// Get variable values from raw variable values
   	double temperature = (ch1 - 1000)/10.0;
   	double humidity = (ch2 - 1000)/10.0;
   	// Convert nowTime to number of ms since 1070-01-01 00:00:00 before getting dateTime
   	Date dateTime = new Date(nowTime*1000);
   	// Get variable value strings from variable values
   	DateFormatter dateFormat = new DateFormatter("yyyy-MM-dd HH:mm:ss");
   	temperatureStr = "" + temperature;
   	humidityStr = "" + humidity;
   	dateTimeStr = dateFormat.format(dateTime);
   }
   catch (ParseException e)
   {
   	System.out.println("JSON ParseException e = " + e);
   }
   

[olle]

Design notes:

1. A server should be set up to regularly collect temperature and humidity values from the different lab sensors, and store the values in a database. As there is no strong dependence between general steps in the lab work and the lab environment data, the latter should be stored in a different database, possibly using a different (simpler) database management system than is used for BASE/Reggie. However, an API should be created to make it easy to retrieve data from the lab environment database, in order to investigate dependencies between individual lab steps in the lab work and the laboratory environment.
2. The lab environment database should store the following data for a lab sensor measurement:
   a. Lab sensor number (uniquely coupled to a sensor URL)
   b. Measurement date/time
   c. Temperature in °C
   d. Humidity in %.
   
3. The tests revealed that the response from a sensor came in a small fraction of a second (of the order of 0.2-0.4 seconds), but the time stamps obtained from the responses of the different sensors could differ more than a minute from each other and the system time of the server, where the test program was executed. In order to simplify investigation of environment measurements from different labs at specific time points, it is recommended that the server system time is used, when a time is stored for a measurement.
4. Tests also revealed that subsequent sensor temperature measurements a minute apart may fluctuate in a range of 0.5 °C, and the humidity measurements in a range of 0.2-0.5 % units, making detailed comparisons of values within this range be of very limited use. When storing lab environment values, it might therefore be considered to collect measurements during a small time interval (some minutes), and then store the arithmetic mean values. An alternative is to leave it up to the routines displaying or otherwise investigating the values, to make any averaging of the raw data.
5. The API for the lab environment database should allow retrieval of data for a chosen sensor number between specified start and end times:
   a. If no start time is given, all data for the chosen sensor before or equal to the end time is retrieved.
   b. If no end time is given, all data for the chosen sensor after or equal to the start time is retrieved.

[olle]

Typo fixed in ticket summary.

[olle]

Design description (initial):

Specific design choices:

• Database management system: The open-source SQLite database management system will be used for storing the lab environment data (see SQLite home page ​https://sqlite.org for more info).
• In order to reduce instrument noise in stored data, and reduce the number of entries in the database, measurements should be collected during a small time interval (some minutes), after which the arithmetic mean values are stored. The time stored together with a stored data value is set to the mean of the times for the measurements used to calculate the average stored data value.
• Packaging: It was decided to create the new service as a BASE extension, allowing for it to be simply installed from a single JAR file plus configuration file using the BASE extension installation mechanism, available through BASE menu "Extensions -> Install plug-ins/extensions...". After installation, the "Extensions" menu will have a new menu item "Lab Environment", with a number of sub-menus. Creating the service as a BASE extension also has the benefit of keeping the code encapsulated, and allowing it to be installed only at BASE servers, that can take advantage of it (i.e. is connected to ThermoWorks Thermo Recoder TR-702W sensors).

The code has been split up in a number of classes/files, where each one has a more or less well-defined responsibility. The code root directory is named "labenv" (short for "lab environment").

Files to be installed in specific directories:

File	Directory	Description
Database file for lab environment values	BASE user files directory defined in configuration file base.config.	File used by the database manager to store the lab environment values. It will be automatically created by the service, if it doesn't exist. Its name is taken from configuration file labenv-config.xml. The BASE user files directory was chosen, as this normally has automatic backup routines coupled to it.
labenv-config.xml	Directory where BASE configuration files like base.config are installed.	This directory was chosen as it is nice to have configuration files stored in a single place.

BASE Service:

File

Description

labenv/src/net/sf/basedb/clients/web/extensions/service/LabEnvService.java

A service based on the BASE DummyAction service, that repeats an action after a pre-defined time interval. This class combines the functionality of ServiceControllerFactory, ServiceController, and Service, that are implemented as separate classes in more complex services, like the BASE FTP Service. It implements the ServiceControllerAction interface, having public methods boolean isRunning(), void start(), and void stop(). Public method void start() creates an instance of singleton LabEnvironment and stores it as an instance variable. LabEnvironment in turn creates an instance of class LabEnvironmentConfiguration, that reads configuration values from file labenv-config.xml. A java.util.TimerTask instance is created, where the time interval is taken from configuration values, and the public void run method calls public method void measureAndStoreLabEnvironmentData() in the LabEnvironment instance.

Lab environment measurements and storage:

File	Description
labenv/src/net/sf/basedb/labenv/LabEnvironment.java	Singleton class that is the main entry point to the lab environment functionality. It creates an instance of class LabEnvironmentConfiguration, that reads configuration values from file labenv-config.xml. Public method void measureAndStoreLabEnvironmentData() uses utility class LabSensorUtil to measure lab environment values for configured lab sensors and store the values using utility class LabEnvironmentStorageUtil.
labenv/src/net/sf/basedb/labenv/LabEnvironmentConfiguration.java	Class that reads configuration values from file labenv-config.xml and stores them in instance variables. Configuration values for individual lab sensors are stored in a list of LabSensorConfig instances. Also has public methods LabSensorConfig findByNumber(Integer number) and LabSensorConfig findByUrl(String url) to obtain the configuration for a specific lab sensor.

Lab environment data access objects:

File	Description
labenv/src/net/sf/basedb/labenv/dao/LabSensorConfig.java	Class for storing lab sensor configuration values in instance variables with corresponding public accessor methods. The values stored are: int number, String url, String name, String startDate, and List<LabSensorAlarmConfig> alarmList.
labenv/src/net/sf/basedb/labenv/dao/LabSensorAlarmConfig.java	Class for storing configuration values for a lab sensor alarm in instance variables with corresponding public accessor methods. For future use.
labenv/src/net/sf/basedb/labenv/dao/LabEnvironmentData.java	Class for storing lab sensor data values in instance variables with corresponding public accessor methods. The values stored are: String sensorUrl, String sensorName, String serialNumber, Date dateTime, long unixTime (number of seconds since 1070-01-01 00:00:00, obtained from dateTime), double temperature (in °C), and double humidity (in %).

Lab environment utility classes:

File	Description
labenv/src/net/sf/basedb/labenv/util/LabSensorUtil.java	Class with public methods for obtaining values from a lab sensor: String fetchLabSensorName(String sensorUrl), LabEnvironmentData createLabEnvironmentData(String sensorUrl), and LabEnvironmentData createLabEnvironmentData(String sensorUrl, String sensorName). Private method String fetchDataFromUrl(String url) is used to obtain the raw sensor output from a sensor at a specified URL.
labenv/src/net/sf/basedb/labenv/util/LabEnvironmentStorageUtil.java	Class for storing and retrieving data in/from the lab environment database. The lab environment database table is created if not existing. The table contains columns for: id INTEGER PRIMARY KEY, sensorNumber numeric, unixTime numeric, temperature numeric, and humidity numeric. Note that sensor measurements are stored by sensor number, not sensor URL or name. The class has public methods ResultSet executeQuery(String instruction), void executeUpdate(String instruction), void updateLabEnvDb(int sensorNumber, LabEnvironmentData envData), void updateLabEnvDb(int sensorNumber, long unixTime, LabEnvironmentData envData), void updateLabEnvDb(int sensorNumber, long unixTime, double temperature, double humidity), and List<LabEnvironmentData> queryLabEnvDb(Integer sensorNumber, Date startTime, Date endTime).
labenv/src/net/sf/basedb/labenv/util/ReportTableUtil.java	Class with utility methods for creating statistics for different time intervals. For future use.

Lab environment servlet and JSP files:

File	Description
labenv/src/net/sf/basedb/labenv/servlet/LabEnvironmentServlet.java	Java servlet class for obtaining information from lab sensors and the lab environment database. Initially three Ajax commands are implemented: GetLabSensorConfigList, LabEnvironmentSensorQuery, and LabEnvironmentDatabaseQuery.
labenv/src/net/sf/basedb/labenv/servlet/LabEnvironmentStatisticsServlet.java	Java servlet class for creating statistics for the lab environment database. Initially two Ajax commands are implemented: labenvironmentdailydistribution and labenvironmentweeklydistribution.
labenv/resources/reports/labsensorinfo.jsp	JSP file for obtaining data from selected lab sensor(s).
labenv/resources/reports/labenvironmentdatabaseexport.jsp	JSP file for obtaining data from the lab environment database, for selected sensor(s) and time period. The obtained data is made available in table format with tab-separated columns, for preview or export to an external file.
labenv/resources/reports/labenvironmentdatabasestatistics.jsp	JSP file for displaying statistics for the lab environment database, for selected sensor(s) and time period. Initially statistics for daily and weekly distribution of temperature and humidity will be available.

BASE extension entry points configuration:

File	Description
labenv/META-INF/extensions.xml	Configuration file for the lab environment BASE extension entry point. Definition of path for extension service class net.sf.basedb.clients.web.extensions.service.LabEnvService, and name of menu item in BASE "Extensions" menu (here a sub-menu with items for individual JSP pages).

Miscellaneous files:

• In addition to the files above, a number of other files are needed to make a working extension. A BASE extension may access core BASE classes, e.g. for obtaining directory paths for configuration and user files, but not utilities in other BASE extensions, such as Reggie. Therefore some practical conversion and formatting utilities have been copied from the latter to the lab environment extension.

Conversion files:

File	Description
labenv/src/net/sf/basedb/labenv/converter/Base64Coder.java	An open-source utility class for coding and decoding data to/from Base-64 format. This class has been used in BASE sister project Proteios SE (​http://www.proteios.org/) since 2006.
labenv/src/net/sf/basedb/labenv/converter/ DateToStringConverter.java StringToDateConverter.java ValueConverter.java	Conversion routines copied from the Reggie extension.

Files for formatting, layout, etc:

File	Description
labenv/resources/css/reggie.css	Copy of Reggie CSS file, ensuring a consistent layout.
labenv/images/	Copy of Reggie "images" directory, ensuring a consistent layout (only a small sub-set of the image files are used).
labenv/resources/reggie.js	Copy of Reggie JavaScript file with practical utility functions for use in web pages.
labenv/src/net/sf/basedb/labenv/LabEnv.java	Small class with static methods for conversion and comparison of values.
labenv/resources/reports/boxplot.js	Copy of Reggie JavaScript file with practical utility functions for creating box plot statistics.

[olle]

Notes on configuration file labenv-config.xml:

Configuration file labenv-config.xml contains the settings for the lab environment measurement and storage. The first version has the following structure:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
	<title>...</title>
	<!--
	<dbfilename>labenv.db</dbfilename>
	<timeoutinseconds>5</timeoutinseconds>
	<storageintervalinseconds>300</storageintervalinseconds>
	<numberofmeasurementsperstoredvalue>5</numberofmeasurementsperstoredvalue>
	-->
	<dbfilename>...</dbfilename>
	<timeoutinseconds>...</timeoutinseconds>
	<storageintervalinseconds>...</storageintervalinseconds>
	<numberofmeasurementsperstoredvalue>...</numberofmeasurementsperstoredvalue>
	<labsensors>
		<labsensor>
			<number>...</number>
			<url>...</url>
			<name>...</name>
			<startdate>...</startdate>
			<alarms>
				<alarm>
					<starttime>...</starttime>
					<endtime>...</endtime>
					<temperaturemin>...</temperaturemin>
					<temperaturemax>...</temperaturemax>        
					<humiditymin>...</humiditymin>
					<humiditymax>...</humiditymax>
					<userlist>
						<user>...</user>
					</userlist>
				</alarm>
			</alarms>
		</labsensor>
		...
	</labsensors>
</parameters>

Global configuration values:

XML tag name	Description
timeoutinseconds	Timeout in seconds when trying to connect to lab sensors.
dbfilename	Name of the database file used to store lab environment data.
storageintervalinseconds	Time interval in seconds between storage of measurement values for lab sensors.
numberofmeasurementsperstoredvalue	Number of measurements performed to get an average value to store. The measurements are performed at times evenly distributed over the time interval between storage of values, e.g. if data values are stored every 300 seconds (5 minutes), and the number of measurements per stored value is 5, a measurement is performed every 300/5 = 60 seconds. The time stored together with a stored data value is set to the mean of the times for the measurements used to calculate the average stored data value.

Configuration values for each lab sensor:

XML tag name	Description
number	Number used to identify the sensor in the lab environment database. It is simplest to think of the sensor number as identifying the location, the sensor is placed in.
url	URL used to access the lab sensor (configured in the internal web server in the physical sensor).
name	Name of the sensor/room the sensor is located in (configured in the internal web server in the physical sensor as the title of the generated web page). This name is used to identify sensors in the sensor selection menu in web pages in the Lab environment BASE extension package.
startdate	A date indicating when the sensor was put in use, and is currently not stored in the database. A sensor with a blank start date will be skipped when measuring and storing lab environment values. This can be used to avoid long connection times before time-out for sensors, that are disconnected from the network during longer time periods.
alarms	An optional list of alarm tags, defining conditions for desired alarms. If triggered, an alarm will be sent to the users in the user list for the alarm in question. (Method of sending alarm is currently not decided.)

An example with two lab sensors is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
	<title>LabEnv options&amp;hellip;</title>
	<!--
	<dbfilename>labenv.db</dbfilename>
	<timeoutinseconds>5</timeoutinseconds>
	<storageintervalinseconds>300</storageintervalinseconds>
	<numberofmeasurementsperstoredvalue>5</numberofmeasurementsperstoredvalue>
	-->
	<dbfilename>labenv.db</dbfilename>
	<timeoutinseconds>5</timeoutinseconds>
	<storageintervalinseconds>300</storageintervalinseconds>
	<numberofmeasurementsperstoredvalue>5</numberofmeasurementsperstoredvalue>
	<labsensors>
		<labsensor>
			<number>1</number>
			<url>http://sensor1.onk.lu.se/</url>
			<name>S1 Instrument room (B32C2)</name>
			<startdate>2014-03-06</startdate>
			<alarms>
			</alarms>
		</labsensor>
		<labsensor>
			<number>2</number>
			<url>http://sensor2.onk.lu.se/</url>
			<name>S2 RNA lab (B33A2)</name>
			<startdate>2014-03-06</startdate>
			<alarms>
			</alarms>
		</labsensor>
	</labsensors>
</parameters>

Measurements from URL's http://sensor1.onk.lu.se/ and http://sensor2.onk.lu.se/ will be stored every 5th minute (5*60 = 300 seconds) in database file "labenv.db". The URL's and sensor names should correspond to the configuration of the internal web servers in the physical sensors. The URL's and names displayed in this example differ from those actually used for the sensors at the Department of Oncology at Lund University, as Swedish terminology is used locally.

• The important thing to note, is that the measurement data will be stored in the database for the sensor number defined in the configuration file. This lab sensor number is therefore not just a trivial index number for internal use in the XML configuration file. When sensors are added for new locations, they should normally be configured with a new unique number. However, if a malfunctioning sensor for an important location needs to be temporarily exchanged for a working sensor internally configured for a another, less important, location, and there is no time to re-configure the latter sensor internally, the lab environment configuration file can be modified so the measurements from the temporary sensor are stored for the sensor number of the malfunctioning one.

[olle]

Ticket Milestone changed to LabEnv v1.0 and component to net.sf.basedb.labenv.

[olle]

(In [2303]) Refs #529. Initial check-in of source code for lab environment BASE extension. The package allows obtaining temperature and humidity values from a number of ThermoWorks Thermo Recoder TR-702W sensors at regular intervals, and storing the data in a special database. The stored data can be exported or directly inspected statistically. The first version allows inspection of the daily or weekly distribution of lab environment values.

[olle]

(In [2304]) Refs #529. BASE jar-files removed from directory labenv/lib/compile/, as it is more practical to add jar-files for desired version separately.

[olle]

(In [2306]) Refs #529. Commit of directory labenv/resources/images/ with small subset of images from Reggie extension, for use in LabEnv extension.

[olle]

(In [2307]) Refs #529. Class/file ReportTableUtil.java in labenv/src/net/sf/basedb/labenv/util/ updated to hopefully fix problem with assigning an hour period string for times on dates when daylight saving time starts or ends (previous algorithm gave the number of hours after midnight):

1. Public method String getCurrentPeriod(Date currentDateTime, String viewType) updated for viewType equal to "dailyDistribution" to use algorithm using SimpleDateFormat(...), that hopefully will be correct also at start and end of daylight saving time.

[olle]

(In [2308]) Refs #529. Class/file ReportTableUtil.java in labenv/src/net/sf/basedb/labenv/util/ updated to reuse SimpleDateformat object :

1. New private instance variable DateFormat hourPeriodDateFormat added together with public accessor method, that creates an instance of SimpleDateFormat("HH"), if hourPeriodDateFormat is null.
2. Public method String getCurrentPeriod(Date currentDateTime, String viewType) updated for viewType equal to "dailyDistribution" to call new public method getHourPeroidDateFormat() to get a DateFormat object to use.

[olle]

Design update:

The initial source code check-in in changeset ​[2303] turned out to have a number of flaws, that will be corrected in sequential updates:

1. Service class LabEnvService should be moved from src/net/sf/basedb/clients/web/extensions/service/ to src/net/sf/basedb/labenv/service/, in order to be a sub-package to net.sf.basedb.labenv, without risk of interfering with outside classes. Configuration file META-INF/extensions.xml should be updated with the change.
2. Jar files commons-httpclient-3.1.jar and sqlite-jdbc-3.7.15-M1.jar should be moved from lib/compile/ to META-INF/lib/, in order to be distributed with the labenv.jar file, as the library files in questions are not included with the BASE distribution. Ant build file build.xml should be updated to include files in META-INF/lib/ in the class path when compiling.
3. Only JAR file slf4j-api-1.6.4.jar is needed in directory lib/compile/, while slf4j-log4j12-1.6.4.jar could be removed.
4. Text files LICENSE, README, and RELEASE should be included in the distribution.
5. The configuration file labenv-config.xml included in the distribution should not include specific values, except for tag numbers for list tag items, needed to make the file valid.
6. The configuration routine should look for the configuration file labenv-config.xml directly, instead of first looking for the BASE configuration file, and then looking for the LabEnv configuration file in the same directory (the reason for the latter procedure was to be able to refer to the directory, the configuration should be located in, in the error message for a missing file). Also, since the functionality of the lab environment extension depends critically on the data in the configuration file labenv-config.xml, an exception should be thrown if no configuration file is found, and the extension should not start.

[olle]

(In [2313]) Refs #529. Update of lab environment service class package:

1. Service class LabEnvService moved from src/net/sf/basedb/clients/web/extensions/service/ to src/net/sf/basedb/labenv/service/, in order to be a sub-package to net.sf.basedb.labenv, without risk of interfering with outside classes.
2. Configuration file META-INF/extensions.xml updated with the change.

[olle]

(In [2314]) Refs #529. Update ensuring that JAR-files not included with the BASE distribution, should be included in the LabEnv distribution:

1. Jar files commons-httpclient-3.1.jar and sqlite-jdbc-3.7.15-M1.jar moved from lib/compile/ to META-INF/lib/, in order to be distributed with the labenv.jar file, as the library files in questions are not included with the BASE distribution.
2. Text file readme.txt in lib/compile/ updated with the removal of files commons-httpclient-3.1.jar and sqlite-jdbc-3.7.15-M1.jar from the directory.
3. Ant build file build.xml updated to include files in META-INF/lib/ in the class path when compiling.

[olle]

(In [2315]) Refs #529. Update of JAR files included for compiling in directory lib/compile/:

1. JAR file slf4j-log4j12-1.6.4.jar removed from directory lib/compile/, since only JAR file slf4j-api-1.6.4.jar is needed.
2. Text file readme.txt in lib/compile/ updated with the removal of file slf4j-log4j12-1.6.4.jar from the directory.

[olle]

(In [2317]) Refs #529. First version of text files LICENSE, README, and RELEASE added.

[olle]

(In [2318]) Refs #529. Configuration file labenv-config.xml included in the distribution updated to not include specific values, except for tag numbers for list tag items, needed to make the file valid.

[olle]

(In [2321]) Refs #529. Lab environment extension updated to look directly for configuration file labenv-config.xml, and not start the service if it could not be found:

1. Class/file LabEnvironmentConfiguration.java in src/net/sf/basedb/labenv updated in private method void fetchConfiguration() to look directly for configuration file labenv-config.xml, and to forward any exception thrown.
2. Class/file LabEnvironment.java in src/net/sf/basedb/labenv updated to forward any exception thrown by the LabEnvironmentConfiguration instance created.
3. Class/file LabEnvironmentServlet.java in src/net/sf/basedb/labenv/servlet/ updated in private method JSONObject createLabSensorConfigList(JSONObject json) to forward any exception thrown by the LabEnvironment instance recast as a ServletException.
4. Class/file LabEnvironmentStatisticsServlet.java in src/net/sf/basedb/labenv/servlet/ updated in private methods JSONArray createLabEnvironmentDailyDistributionReport(JSONArray jsonStatisticsPlotArray, List<LabEnvironmentData> lthdList, Date startDate, Date endDate, String chartVariant, String viewType, String sensorName, String weekdayFilter) and String fetchChartTitle(String viewType, String variableType, Date startDate, Date endDate, String weekdayFilter) to forward any exception thrown by the LabEnvironment instance recast as a ServletException.
5. Class/file LabEnvironmentStorageUtil.java in src/net/sf/basedb/labenv/util/ updated in public constructor LabEnvironmentStorageUtil() and public method List<LabEnvironmentData> queryLabEnvDb(Integer sensorNumber, Date startTime, Date endTime) to catch any exception thrown by the LabEnvironment instance and print its content to standard output.

[olle]

Design update:

• Update in changeset ​[2321] is unsatisfactory in two ways:
  1. An exception thrown when trying to start the extension is not reported in the BASE window, only in the standard output.
  2. The Tomcat server needs to be restarted, if one has changed the configuration file labenv-config.xml, and wants it to be reloaded; just restarting the extension is not enough.
  

Possible remedies:

1. A RuntimeException() can be thrown by public method void start() in class LabEnvService, without upsetting the ServiceControllerAction Interface. By also letting private method void fetchConfiguration() in class LabEnvironmentConfiguration forward the contents of a caught exception as a RuntimeException, one doesn't have to declare that a number of methods throw exceptions.
2. A new public method void reloadLabEnvironmentConfiguration() could be added to singleton class LabEnvironment, for forcing a reload of the configuration file. Class LabEnvService could then be updated in public method void start() to call reloadLabEnvironmentConfiguration(), to ensure that the configuration file is read, when the service is started.

[olle]

(In [2326]) Refs #529. Lab environment extension updated to report in the BASE window, when configuration file labenv-config.xml is missing, in which case the extension cannot start. Also, the configuration file is now reloaded every time the extension starts, even if the Tomcat server isn't restarted:

1. Singleton class/file LabEnvironment updated with new new public method void reloadLabEnvironmentConfiguration(), for forcing a reload of the configuration file.
2. Class/file LabEnvironmentConfiguration updated in private method void fetchConfiguration() to forward the contents of a caught exception as a RuntimeException, avoiding the need to declare that a number of methods in other classes throw exceptions.
3. Class/file LabEnvService.java in src/net/sf/basedb/labenv/service/ updated in public method void start() to call new public method void reloadLabEnvironmentConfiguration() in class LabEnvironment, and to forward the contents of a caught exception as a RuntimeException, which doesn't violate the ServiceControllerAction Interface.
4. Class/file LabEnvironmentServlet.java in src/net/sf/basedb/labenv/servlet/ updated in private method JSONObject createLabSensorConfigList(JSONObject json) to no longer forward exceptions thrown by the LabEnvironment instance recast as a ServletException.
5. Class/file LabEnvironmentStatisticsServlet.java in src/net/sf/basedb/labenv/servlet/ updated in private methods JSONArray createLabEnvironmentDailyDistributionReport(JSONArray jsonStatisticsPlotArray, List<LabEnvironmentData> lthdList, Date startDate, Date endDate, String chartVariant, String viewType, String sensorName, String weekdayFilter) and String fetchChartTitle(String viewType, String variableType, Date startDate, Date endDate, String weekdayFilter) to no longer forward exceptions thrown by the LabEnvironment instance recast as a ServletException.
6. Class/file LabEnvironmentStorageUtil.java in src/net/sf/basedb/labenv/util/ updated in public constructor LabEnvironmentStorageUtil() and public method List<LabEnvironmentData> queryLabEnvDb(Integer sensorNumber, Date startTime, Date endTime) to no longer catch exceptions thrown by the LabEnvironment instance and print its content to standard output.

[olle]

(In [2328]) Refs #529. Lab environment extension updated to increase similarity of logging with that used by BASE. Package org.apache.log4j.Logger exchanged for org.slf4j.Logger and org.slf4j.LoggerFactory. Logger object properties changed from protected static to private static final. Affected files:

1. JAR file log4j-1.2.16.jar removed from lib/compile/.
2. Class/file LabEnvironment.java in src/net/sf/basedb/labenv/ updated.
3. Class/file LabEnvironmentConfiguration.java in src/net/sf/basedb/labenv/ updated.
4. Class/file LabEnvService.java in src/net/sf/basedb/labenv/service/ updated.
5. Class/file LabEnvironmentServlet.java in src/net/sf/basedb/labenv/servlet/ updated.
6. Class/file LabEnvironmentStatisticsServlet.java in src/net/sf/basedb/labenv/servlet/ updated.
7. Class/file LabEnvironmentStorageUtil.java in src/net/sf/basedb/labenv/util/ updated.
8. Class/file LabSensorUtil.java in src/net/sf/basedb/labenv/util/ updated.

[olle]

(In [2329]) Refs #529. Lab environment extension updated in classes forwarding exception as a RuntimeException to forward the full exception information, and not just the message:

1. Class/file LabEnvironmentConfiguration in src/net/sf/basedb/labenv/ updated in private method void fetchConfiguration() to forward the full information of a caught exception as a RuntimeException.
2. Class/file LabEnvService in src/net/sf/basedb/labenv/service/ updated in public method void start() to forward the full information of a caught exception as a RuntimeException.

[olle]

(In [2330]) Refs #529. Class/file LabEnvironmentConfiguration in src/net/sf/basedb/labenv/ updated in private method void fetchConfiguration() to throw a RuntimeException with an appropriate message, if configuration file labenv-config.xml could not be found. This should result in a better error message, if the configuration file is missing, the first time the extension is started.

[olle]

(In [2331]) Refs #529. Class/file LabEnvironmentConfiguration in src/net/sf/basedb/labenv/ updated in private method void fetchConfiguration() by fix of bug when forwarding the full information of a caught exception as a RuntimeException.

[olle]

(In [2332]) Refs #529. Lab environment extension updated in file MANIFEST.MF in META-INF/ by adding files commons-httpclient-3.1.jar and sqlite-jdbc-3.7.15-M1.jar distributed with extension in directory META-INF/lib/ to class path.

[olle]

(In [2337]) Refs #529. Lab environment extension updated to use default values where possible, if no configuration values are supplied in configuration file labenv-config.xml, or a configuration value is out-of-bounds:

1. Configuration file labenv-config.xml updated:
   a. Database filename setting is put before settings for communicating with the sensors.
   b. A comment block with default values for database filename, timeout in seconds, storage interval in seconds, and number of measurements per stored value is inserted before the configuration lines for these variables.
   
2. Class/file LabEnvironmentConfiguration.java in src/net/sf/basedb/labenv/ updated in private method void fetchConfiguration() to use default values for database filename, timeout in seconds, storage interval in seconds, and number of measurements per stored value, if no configuration values are supplied in configuration file labenv-config.xml, or a configuration value is out-of-bounds. Code for settings of configuration values moved to came in the same order as the values in the configuration file.
3. Class/file LabEnvironmentStorageUtil.java in src/net/sf/basedb/labenv/util/ updated to not set a default database filename, since that is now handled by class LabEnvironmentConfiguration.

[olle]

(In [2338]) Refs #529. Class/file LabEnvService.java in src/net/sf/basedb/labenv/service/ updated with new public method String toString(), returning service name "LabEnv Server extension". This string is e.g. shown in the window opened from BASE menu Administrate -> Services.

[olle]

(In [2341]) Refs #529. JSP files labsensorinfo.jsp, labenvironmentdatabaseexport.jsp, and labenvironmentdatabasestatistics.jsp in resources/reports/ updated to set link for header text "LabEnv" to the same JSP page, as currently the lab environment extension doesn't have a common index page.

[olle]

(In [2342]) Refs #529. Lab environment extension updated by replacing all remaining System.out.println() calls with logging using org.slf4j.Logger and org.slf4j.LoggerFactory (these calls are used in LabEnvService to log that the service is started or stopped). Earlier calls to System.out.println() that has been commented out are now removed. Also removal of some other code lines, that has been commented out. Affected files:

1. Class/file LabEnvironment.java in src/net/sf/basedb/labenv/ updated.
2. Class/file LabEnvironmentConfiguration.java in src/net/sf/basedb/labenv/ updated.
3. Class/file LabEnvService.java in src/net/sf/basedb/labenv/service/ updated.
4. Class/file LabEnvironmentServlet.java in src/net/sf/basedb/labenv/servlet/ updated.
5. Class/file LabEnvironmentStatisticsServlet.java in src/net/sf/basedb/labenv/servlet/ updated.
6. Class/file LabEnvironmentStorageUtil.java in src/net/sf/basedb/labenv/util/ updated.
7. Class/file LabSensorUtil.java in src/net/sf/basedb/labenv/util/ updated.

[olle]

Design discussion:

• Re-loading the LabEnv extension sometimes leads to a java.lang.UnsatisfiedLinkError: org.sqlite.NativeDB._open(Ljava/lang/String;I)V error, due to multiple copies of the sqlite JAR file. It is therefore recommended that the sqlite-jdbc-3.7.15-M1.jar file is copied to the Tomcat lib directory, instead of being distributed with the extension in its META-INF/lib/ directory. JAR file sqlite-jdbc-3.7.15-M1.jar can be downloaded from ​https://bitbucket.org/xerial/sqlite-jdbc/downloads/. If the error should still occur, the problem can normally be fixed by restarting Tomcat.

[olle]

(In [2345]) Refs #529. Lab environment extension updated by moving JAR file sqlite-jdbc-3.7.15-M1.jar back from META-INF/lib/ to lib/compile/, in order to try to avoid a java.lang.UnsatisfiedLinkError: org.sqlite.NativeDB._open(Ljava/lang/String;I)V error, due to multiple copies of the sqlite JAR file. JAR file sqlite-jdbc-3.7.15-M1.jar therefore should be copied to the Tomcat lib directory, instead of being distributed with the extension. (The JAR file can be downloaded from ​https://bitbucket.org/xerial/sqlite-jdbc/downloads/):

1. JAR file sqlite-jdbc-3.7.15-M1.jar removed from directory META-INF/lib/.
2. File MANIFEST.MF in META-INF/ updated by removing file sqlite-jdbc-3.7.15-M1.jar from the class path.
3. JAR file sqlite-jdbc-3.7.15-M1.jaradded to directory lib/compile/.
4. Text file README updated in the installation instructions to include directions on downloading JAR file sqlite-jdbc-3.7.15-M1.jar from ​https://bitbucket.org/xerial/sqlite-jdbc/downloads/, and placing it in the Tomcat lib/` directory.

• If the error should still occur, the problem can normally be fixed by restarting Tomcat.

[olle]

(In [2346]) Refs #529. Lab environment extension updated in outermost Ant build file build.xml for target "package":

1. Entries for files labenv.jar (via ${jar.name}), README, and LICENSE* removed from tarfileset for directory ".", since these are already included in target "dist", that target "package" depends on.
2. A new tarfileset for directory "lib/compile" added, in order to include jar file sqlite-jdbc*.jar in the labenv-*.tar.gz package.

[olle]

(In [2347]) Refs #529. Lab environment extension updated in text file README:

1. Installation instructions updated, as jar file sqlite-jdbc*.jar now is included in the labenv-*.tar.gz package, and does not have to be downloaded.

[olle]

(In [2349]) Refs #529. Lab environment extension updated to only output 2 decimals when exporting temperature and humidity data:

1. Data access object class/file LabEnvironmentData in src/net/sf/basedb/labenv/dao/ updated in public method String toString() to only output 2 decimals for temperature and humidity data.

[olle]

(In [2352]) Refs #529. Lab environment extension updated when exporting temperature and humidity data:

1. Data access object class/file LabEnvironmentData in src/net/sf/basedb/labenv/dao/ updated in public method String toString() for temperature and humidity data, with removal of unnecessary string concatenation.

[olle]

(In [2353]) Refs #529. Lab environment extension updated in text file readme.txt in lib/compile/:

1. Text file readme.txt in lib/compile/ updated with inclusion of file sqlite-jdbc-3.7.15-M1.jar, as the latter was placed back into directory lib/compile/ in changeset ​[2345].

[Nicklas Nordborg]

(In [2354]) References #529. Add eclipse project and classpath file and svn:ignore properties.

[olle]

(In [2357]) Refs #529. Configuration file META-INF/extensions.xml updated by setting version to "1.0" and min-base-version to "3.2.2".

[olle]

Ticket closed as a first version of the lab environment extension has been created.