net.sf.basedb.normalizers: Normalizers should use ask BASE whether data is logged or not

[Jari Häkkinen]

and use that information appropriately. Either set a proper default selection of average to use or simply use proper averaging without asking the user how to average.

Logged values should use arithmetic averages by default, non-logged should use geometric average by default.

Adding above functionality requires BASE 2.12 use.

[Martin Svensson]

(In [1143]) References #228 Added parameter to define which average method to use when running the plugin. The functionality is not tested yet.

[Martin Svensson]

(In [1144]) References #228 Updated the method that calculates bioassayset's global average in AverageNormalizationPlugin, to support userdefine average method. Documentation related to recent changes has also been updated. The updated plugins execute as expected but the normalized data has not yet been verified.

[olle]

Ticket accepted.

[olle]

Background discussion (Thanks to Nicklas Nordborg for detailed information about this):

1. Originally data in BASE was stored in untransformed format, and the average method used was arithmetic mean (the latter differs from the current recommendation for untransformed data, which is to use geometric mean).
2. A flag indicating if stored data was untransformed, stored as log-2, or log-10 values, was introduced in BASE Ticket ​#1120 (The dynamic part of BASE should keep track whether intensity data is in log space or not). See this ticket for a lengthy discussion on the use and storage of transformed data in BASE. Some additions were:
   
   a. A BioAssaySet has a getIntensityTransform() method that returns an IntensityTransform enum object, whose public Formula.AverageMethod getAverage() method returns Formula.AverageMethod.ARITHMETIC_MEAN or Formula.AverageMethod.GEOMETRIC_MEAN.
   b. Class VirtualColumn was extended with static methods VirtualColumn channelIntensity(int channel) and VirtualColumn channelRaw(int channel). The former performs a reverse transformation if needed, determined by the transformation flag, in order to return untransformed data, while the latter returns data the way it was stored in the database, i.e. data in log-2 and log-10 format is returned in that format.
   
   
3. Enum class IntensityTransform public Formula.AverageMethod getAverage() method does however return Formula.AverageMethod.ARITHMETIC_MEAN for data stored in untransformed format (according to the flag), and Formula.AverageMethod.GEOMETRIC_MEAN for data stored in log-2 or log-10 format. This is the opposite to what is recommended to be used in this ticket, for data fetched in untransformed form.

[olle]

Traceability note:

• BASE Ticket ​#1120 (The dynamic part of BASE should keep track whether intensity data is in log space or not) added several new methods and items concerning work with transformed/untransformed data.
• Ticket #541 (net.sf.basedb.normalizers: Normalizers should store normalized data in proper base) is concerned with how data should be stored after normalization.

[olle]

Problem discussion:

• The desired default option for averaging method can be obtained by switching the results returned by public method Formula.AverageMethod getAverage() in enum class IntensityTransform for data flagged to be stored in untransformed, relative to transformed (log-2 or log-10) format:
  
  Formula.AverageMethod.GEOMETRIC_MEAN for data stored in untransformed format
  Formula.AverageMethod.ARITHMETIC_MEAN for data stored in transformed format
  
  However, since the gerAverage() method is used in several other classes, a thorough investigation is needed before such a change should be added, since there is a risk that functionality is broken in these other classes.

[olle]

Problem discussion update:

• BASE Ticket ​#1792 (Incorrect average method specified in IntensityTransform) switched the results returned by public method Formula.AverageMethod getAverage() in enum class IntensityTransform for data flagged to be stored in untransformed, relative to transformed (log-2 or log-10) format (see changeset ​[6564]):
  
  Formula.AverageMethod.GEOMETRIC_MEAN for data stored in untransformed format
  Formula.AverageMethod.ARITHMETIC_MEAN for data stored in transformed format
  
  This would solve the problem with wrong default averaging method being shown for QuantileNormalizer. However, there are benefits letting the plug-in itself determine the default averaging method, based on the intensity transform information for the BioAssaySet to work on.
  
  Class IntensityTransform was extended with new public method double transform(double value) in changeset ​[6365], that stores data in the same transform as the source data.

[olle]

(In [2154]) Refs #228. Classes/files AverageNormalization.java and QuantileNormalization.java updated to determine the default averaging method based on the intensity transform information for the BioAssaySet to work on, instead of what is returned from the Formula.AverageMethod getAverage() method for the IntensityTransform of the latter:

1. Private method RequestInformation getConfiguredJobParameters() is updated to check the intensity transform set for the BioAssaySet, and select the default averaging method based on that. This makes the choice of default averaging method independent on the value of the getAverage() method.

[olle]

Design update for QuantileNormalization:

1. Data stored in logarithmic format should be untransformed before averaging, and then transformed back to logarithmic format before results are stored. The first part is already implemented by use of static method VirtualColumn channelIntensity(int channel) in class AbstractNormalizationPlugin, while public method double transform(double value) in class IntensityTransform should be used to transform the normalized result back before storing results.
2. Since averaging is performed on untransformed data, default averaging method should always be set to Formula.AverageMethod.GEOMETRIC_MEAN.
3. Help text for selecting averaging method in class AbstractNormalizationPlugin should be updated to avoid references to the format data is stored in, since averaging is performed on untransformed data.
4. Help text for Quantile normalization in META-INF/extensions.xml should be extended with information that data stored in logarithmic format will be untransformed before averaging, and then transformed back to logarithmic format before results are stored.

[olle]

(In [2165]) Refs #228. Refs #541. Quantile normalization updated to untransform logarithmic data before normalization, and then transform it back to logarithmic format before results are stored. Default averaging method will always be set to Formula.AverageMethod.GEOMETRIC_MEAN:

1. Class/file QuantileNormalization.java in src/net/sf/basedb/plugins/ in package net.sf.basedb.normalizers updates:
   a. Private method RequestInformation getConfiguredJobParameters() updated to always set the default averaging method to Formula.AverageMethod.GEOMETRIC_MEAN.
   b. Private method BioAssaySet normalize(DbControl dc, BioAssaySet source, Job job, ProgressReporter progress) updated to call public method double transform(double value) in class IntensityTransform to transform the normalized result back before storing it. Error message when number of spots differ between two BioAssay sets updated to report the names of the latter and the number of spots in each one. Also minor updates in order to increase clarity of code.
2. Class/file AbstractNormalizationPlugin.java in src/net/sf/basedb/plugins/ in package net.sf.basedb.normalizers updated in help text for selecting averaging method by avoiding reference to the format data is stored in, since averaging is performed on untransformed data.
3. XML files extensions.xml in META-INF in package net.sf.basedb.normalizers updated in help text for Quantile normalization by adding information that data stored in logarithmic format will be untransformed before averaging, and then transformed back to logarithmic format before results are stored.

[olle]

Design update for AverageNormalization (mirrors update for QuantileNormalization):

1. Data stored in logarithmic format should be untransformed before averaging, and then transformed back to logarithmic format before results are stored. The first part is already implemented by use of static method VirtualColumn channelIntensity(int channel) in class AbstractNormalizationPlugin, while public method double transform(double value) in class IntensityTransform should be used to transform the normalized result back before storing results.
2. Since averaging is performed on untransformed data, default averaging method should always be set to Formula.AverageMethod.GEOMETRIC_MEAN.

[olle]

(In [2166]) Refs #228. Refs #541. Average normalization updated to untransform logarithmic data before normalization, and then transform it back to logarithmic format before results are stored. Default averaging method will always be set to Formula.AverageMethod.GEOMETRIC_MEAN:

1. Class/file AverageNormalization.java in src/net/sf/basedb/plugins/ in package net.sf.basedb.normalizers updates:
   a. Private method RequestInformation getConfiguredJobParameters() updated to always set the default averaging method to Formula.AverageMethod.GEOMETRIC_MEAN.
   b. Private method BioAssaySet normalize(DbControl dc, BioAssaySet source, Job job, float refValue, float minIntensity, ProgressReporter progress) updated to call public method double transform(double value) in class IntensityTransform to transform the normalized result back before storing it.

[olle]

Design note:

• When data stored in logarithmic format are untransformed before averaging, and then transformed back to logarithmic format before results are stored, transformation information for the result BioAssaySet must be set explicitly by calling its public void setIntensityTransform(IntensityTransform transform) method. If this is not done, methods using the result BioAssaySet will treat the data in logarithmic format as original data.

[Jari Häkkinen]

(In [2173]) Refs #228 and #541. Improving doc and help string

[olle]

(In [2174]) Refs #541. Refs #228. Outermost Ant build XML file updated by increasing dependency base-version from 3.0.0 to 3.2.4, as this is required to use stored information on input data format (pristine, log-2, or log-10).

[olle]

Ticket closed as the requested functionality has been added.

[Nicklas Nordborg]

Milestone renamed