#461 closed enhancement (fixed)
Sample processing report generator
Reported by: | olle | Owned by: | olle |
---|---|---|---|
Priority: | major | Milestone: | Reggie v2.11 |
Component: | net.sf.basedb.reggie | Keywords: | |
Cc: |
Description (last modified by )
Reggie should be updated to include a sample processing report generator, producing a report similar to the one produced today for the SCAN-B project using an R script. The latter report consists of a number of box plots showing the statistical distribution of various quantities related to the sample processing. The box plots show the statistics per quarter and per month for the selected time period.
Change History (39)
comment:1 by , 12 years ago
Status: | new → assigned |
---|
comment:2 by , 12 years ago
Traceability note:
- The report generator was introduced in Ticket #339 (Report generator). It included a sample count report.
- A consent count report was introduced in Ticket #426 (Consent count report generator).
- A patient count report was introduced in Ticket #433 (Patient count report generator).
- An overview report was introduced in Ticket #438 (Overview report).
- A missing sample data report was introduced in Ticket #439 (Missing sample data report).
- Common table report utilities was broken out from the sample count report code and placed in its own class in Ticket #459 (Common table report utilities should be placed in its own class).
comment:3 by , 12 years ago
User interface design:
- The previous reports (sample count, consent count, patient count, overview, and missing sample data) were all placed under the sub-header "Report generator". Since they are all concerned with the original samples and their sources, this sub-header will be renamed "Sample source report".
- The new report will be named "SCAN-B quarter/month report", and placed under a new sub-header "Sample processing report".
- The interface for the SCAN-B quarter/month report will be based on that for the previous reports, with three modifications:
1. The view type selection will be extended with an option "Quarter + Month", that will produce plots for the selected quantities first by quarter and then by month.
2. The view type "Auto" option will never return "Week", since this is normally too short a period for items of interest in sample processing. However, the user may still be able to manually select "Week" from the view type select box.
3. A new selection box "Chart data" will be added, with options for all of the quantities, for which plots were produced by the original R script. An option "All" will also be available (and the default), that will produce plots for all of the quantities. If view type is set to "Quarter + Month", and chart data to "All", first plots will be displayed for all quantities per quarter, and then by month.
comment:4 by , 12 years ago
Description: | modified (diff) |
---|
Added basic info on the R script box plots.
comment:6 by , 12 years ago
Short description of the original sample processing report (produced by the R script):
- The report contains quarterly and monthly statistics for the following quantities:
a. Original quantity used for SCANB specimens
b. Quantity tissue used for SCANB specimens
c. Histology piece quantity used for SCANB specimens
d. Remaining tissue quantity for SCANB specimens
e. Total quantity DNA used for SCANB extractions
f. Total quantity RNA used for SCANB extractions
g. DNA yield (µg/mg tissue) SCANB
h. RNA yield (µg/mg tissue) SCANB
i. DNA yield corrected for lysate volume (µg/mg tissue) SCANB
j. RNA yield corrected for lysate volume (µg/mg tissue) SCANB
k. RNA QC SCANB, RQS otherwise RQS~RIN
l. Minutes to RNAlater - The R script uses input from two files, containing data for samples and extracts, respectively. Samples of subtypes
Specimen
,Histology
, andCase
are used, and extracts of subtypesDNA
,RNA
,Lysate
, andRNAQC
. - The R script discards samples with no known original quantity and samples from biopsies.
- The R script creates box plots for the statistics of each quantity, where the center box in the box plot is determined by the 25- and 75-percentile, and contains a marker for the 50-percentile (the median). The whiskers on each box show the smallest and largest data values that lie within a distance of 1.5 times the value distance between the 25- and 75-percentile (the inter-percentile range, IPR) from the bottom and top of the box.
Comments: RQS stands for "RNA Quality Score" and RIN for "RNA Integrity Number". RQS is used on the Caliper instrument, while RIN is used on the BioAnalyzer.
comment:7 by , 12 years ago
Design description.
A preliminary version of the SCAN-B quarter/month report in Reggie will be based on the following components:
- A servlet class/file
ScanBQuarterMonthReportServlet.java
inreggie/src/net/sf/basedb/reggie/servlet/
. The servlet will perform the main data processing and calculate the statistics. This has the advantage of reducing the bandwidth need by only having to send the statistics values to the receiving JavaScript in a JSON object, instead of the whole data set, that would have been necessary if the statistics calculation were performed by the script. - A JSP script
scanbquartermonthreportgenerator.jsp
inreggie/resources/
. The JSP script will manage the GUI and display the created box plots. - A JavaScript utility file
boxplot.js
inreggie/resources/
. The utility file will take statistics data in JSON format as input and create a boxplot in an HTML 5 canvas element. - JSP script
index.jsp
inreggie/resources/
is updated in section "Reggie reports" by having sub-header "Report generator" renamed "Sample source report" and addition of a new sub-header "Sample processing report", with an item "SCAN-B quarter/month report".
Statistics calculation:
- Percentiles will be calculated by linear interpolation between values for nearest items, in case no item exactly corresponds to the percentile.
- Statistics for a quantity will only be performed if at least 5 items exist for the time period in question.
In order to create the statistics, a quantity value and an associated date must be selected for each chart type. In some cases, the choice is straight forward, while in other more than one option is available. The following table shows the quantity values and dates that are used in the first preliminary version of the report. The status column shows a comment on how well the generated plot and displayed data reproduce the printed plots created by the original procedure using the R script.
Chart data | Quantity used | Date used | Status |
Original tissue quantity | sample.getOriginalQuantity()/1000.0f | Annotation value QIACUBE_DATE of extract from lysate from sample | OK |
Quantity tissue used | extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample (may be more than one extract/sample) | Annotation value QIACUBE_DATE of extract from lysate from sample | OK |
Histology piece quantity | sample.getOriginalQuantity()/1000.0f for histology sample | Annotation value PARTITION_DATE of histology sample | Extra items, partial resemblance |
Remaining tissue quantity | sample.getRemainingQuantity()/1000.0f | Annotation value QIACUBE_DATE of extract from lysate from sample | OK |
Total quantity DNA used | extract.getOriginalQuantity() for DNA extract | Annotation value QIACUBE_DATE of DNA extract | OK |
Total quantity RNA used | extract.getOriginalQuantity() for RNA extract | Annotation value QIACUBE_DATE of RNA extract | OK |
DNA yield (µg/mg tissue) | extract.getOriginalQuantity() for DNA extract divided by extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample | Annotation value QIACUBE_DATE of DNA extract | OK |
RNA yield (µg/mg tissue) | extract.getOriginalQuantity() for RNA extract divided by extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample | Annotation value QIACUBE_DATE of RNA extract | OK |
DNA yield corrected for lysate volume (µg/mg tissue) | 2 x extract.getOriginalQuantity() for DNA extract divided by extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample | Annotation value QIACUBE_DATE of DNA extract | Resemblance OK |
RNA yield corrected for lysate volume (µg/mg tissue) | 2 x extract.getOriginalQuantity() for RNA extract divided by extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample | Annotation value QIACUBE_DATE of RNA extract | Resemblance OK |
RNA QC SCANB, RQS otherwise RQS~RIN | Annotations CA_RQS and BA_RIN for RNA extract. When no RQS value exists, the value RIN*0.7698785 + 1.607572 is used. | Annotation value QIACUBE_DATE of RNA extract | Missing items, order of magnitude OK |
Minutes to RNAlater | Difference between annotation values RNALATER_DATETIME and SAMPLING_DATETIME for specimen sample, converted to minutes | Annotation value QIACUBE_DATE of extract from lysate from sample | Extra items, order of magnitude OK |
comment:8 by , 12 years ago
(In [1820]) Refs #461. Preliminary version of SCAN-B quarter/month report:
- XML file
servlets.xml
inreggie/META-INF/
updated with entry for new servletnet.sf.basedb.reggie.servlet.ScanBQuarterMonthReportServlet
. - JSP file
index.jsp
inreggie/resources/
updated by renaming of entry "Report generator" to "Sample source report" and adding new entry "Sample processing report. - JavaScript file
boxplot.js
inreggie/resources/
added. It contains functions for producing simple box plots in an HTML canvas element from input data in a JSON object. - JSP file
samplereportgenerator.jsp
inreggie/resources/
updated by renaming of header "Report generator" to "Sample source report". - JSP file
scanbquartermonthreportgenerator.jsp
added. It manages the GUI for the SCAN-B quarter/month report. - Java class/file
ScanBQuarterMonthReportServlet.java
inreggie/src/net/sf/basedb/reggie/servlet/
added. It collects the data and performs the statistical calculations for the report.
comment:9 by , 12 years ago
comment:10 by , 12 years ago
comment:11 by , 12 years ago
comment:12 by , 12 years ago
comment:13 by , 12 years ago
comment:14 by , 12 years ago
(In [1837]) Refs #461. Java class/file ScanBQuarterMonthReportServlet.java
in reggie/src/net/sf/basedb/reggie/servlet/
updated in report for min to RNAlater:
- Samples where time of day for sampling or RNAlater equals 00:00:00 are now excluded.
- Samples where min to RNAlater is 0 or negative are now included.
- Name, sampling date & time, and RNAlater date & time are now reported in a list for samples where min to RNAlater is 0 or negative.
comment:15 by , 12 years ago
(In [1840]) Refs #461. Java class/file ScanBQuarterMonthReportServlet.java
in reggie/src/net/sf/basedb/reggie/servlet/
updated in report for histology piece quantity:
- The date used for a histology sample is change from the partition date to the QiaCube date of extract from lysate from parent sample of histology sample.
comment:16 by , 12 years ago
(In [1842]) Refs #461. Java class/file ScanBQuarterMonthReportServlet.java
in reggie/src/net/sf/basedb/reggie/servlet/
updated in report for RNA QC, plus other minor changes:
- RNA QC report updated by complete rewriting of code. Previous code contained two major errors, checks were made against a list that was continuously updated, and the RQS value was erroneously used as the RIN value. The updated code has the following major changes:
a. RQS and RIN values are checked to be > 0 (-100 is used as a flag for bad data).
b. The selection process is now based on the grand parent sample, i.e. only one RNA QC value is used for a single sample. If a valid RQS value exists, it is used, otherwise a valid RIN value. - The number of decimals used in reported values has been set to 1.
- Removal of variables used for test and debugging purposes.
comment:17 by , 12 years ago
(In [1843]) Refs #461. Sample processing report is updated to allow appended text after the plot section:
- JSP file
scanbquartermonthreportgenerator.jsp
inreggie/resources/
updated by retrieving optional text to append from JSON object with key "appendedInfo
". - Java class/file
ScanBQuarterMonthReportServlet.java
inreggie/src/net/sf/basedb/reggie/servlet/
updated to add information on number of samples with negative or zero min to RNAlater value as appended info text. This appended info is only added if "min to RNAlater" is included in the selected chart data.
comment:18 by , 12 years ago
(In [1845]) Refs #461. Boxplot routines for drawing y-axis scale markers now separates between the value used for placing the marker, and the text displayed at the marker. Besides allowing general text to be used, this solves a problem when values were modified to a fixed number of decimals for display, which affected the placement of the marker in the previous design. Now a precise value should be used for placement of the marker, and a truncated value may be used as display text:
- JavaScript file
boxplot.js
inreggie/resources/
updated in functionsdrawScaleMarkerYLeft(...)
anddrawScaleMarkerYRight(...)
with new argumentmarkerText
, for the text to be displayed at the marker. FunctioncreateBoxPlot(boxPlotJsonObject, ...)
updated to retrieve the text to displayed at the markers for dotted horizontal guidelines from the JSON object for guide lines. - Java class/file
ScanBQuarterMonthReportServlet.java
inreggie/src/net/sf/basedb/reggie/servlet/
updated with two new private convenience methodsJSONObject createValueWithText(Float value, int numberOfDecimalsShown)
andJSONObject createValueWithText(Float value, String text)
. Both creates a JSONObject with keys "value
" and "text
". Private methodJSONObject createJSONPlotStatistics(...)
updated to use new methodJSONObject createValueWithText(Float value, int numberOfDecimalsShown)
when creating JSONObjects for the dotted horizontal guidelines.
comment:19 by , 12 years ago
Revised overview of the SCAN-B quarter/month report in Reggie:
Chart data | Quantity used | Date used | Status |
Original tissue quantity | sample.getOriginalQuantity()/1000.0f | Annotation value QIACUBE_DATE of extract from lysate from sample | OK |
Quantity tissue used | extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample (may be more than one extract/sample) | Annotation value QIACUBE_DATE of extract from lysate from sample | OK |
Histology piece quantity | sample.getOriginalQuantity()/1000.0f for histology sample | Annotation value QIACUBE_DATE of extract from lysate from parent sample of histology sample | OK |
Remaining tissue quantity | sample.getRemainingQuantity()/1000.0f | Annotation value QIACUBE_DATE of extract from lysate from sample | OK |
Total quantity DNA used | extract.getOriginalQuantity() for DNA extract | Annotation value QIACUBE_DATE of DNA extract | OK |
Total quantity RNA used | extract.getOriginalQuantity() for RNA extract | Annotation value QIACUBE_DATE of RNA extract | OK |
DNA yield (µg/mg tissue) | extract.getOriginalQuantity() for DNA extract divided by extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample | Annotation value QIACUBE_DATE of DNA extract | OK |
RNA yield (µg/mg tissue) | extract.getOriginalQuantity() for RNA extract divided by extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample | Annotation value QIACUBE_DATE of RNA extract | OK |
DNA yield corrected for lysate volume (µg/mg tissue) | 2 x extract.getOriginalQuantity() for DNA extract divided by extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample | Annotation value QIACUBE_DATE of DNA extract | OK |
RNA yield corrected for lysate volume (µg/mg tissue) | 2 x extract.getOriginalQuantity() for RNA extract divided by extract.getCreationEvent() .getUsedQuantity(sample)/1000.0f for extracts from sample | Annotation value QIACUBE_DATE of RNA extract | OK |
RNA QC SCANB, RQS otherwise RQS~RIN | Only one value per grand parent sample (sample of lysate of RNA extract of RNA QC extract) is used. Annotations CA_RQS and BA_RIN for RNA parent extract of RNA QC. Only valid RQS and RIN values are used, i.e. value > 0. When no RQS value exists, the value RIN*0.7698785 + 1.607572 is used. | Annotation value QIACUBE_DATE of parent RNA extract | OK |
Minutes to RNAlater | Difference between annotation values RNALATER_DATETIME and SAMPLING_DATETIME for specimen sample, converted to minutes. Entries with time set to "00:00:00" are discarded. | Annotation value QIACUBE_DATE of extract from lysate from sample | OK |
comment:20 by , 12 years ago
(In [1846]) Refs #461. Java class/file ScanBQuarterMonthReportServlet.java
in reggie/src/net/sf/basedb/reggie/servlet/
updated for appended info on special samples if "min to RNAlater" is included in the chart selection:
- The information is now displayed in tables instead of in lists.
- The sample info now includes a link to the case summary for the sample.
comment:21 by , 12 years ago
comment:22 by , 12 years ago
comment:23 by , 12 years ago
(In [1849]) References #462: Implement a print function that can print complete pages
Implemented a javascript function that opens a popup window and copies the content of a specified html tag (given by the id) to the popup window.
openPrintWindow(ID, printElementId, pageTitle, pageOrientation, printNote)
The change also includes printing changes made in separate branch for #425 in [1723].
comment:24 by , 12 years ago
Note: The update of JSP file scanbquartermonthreportgenerator.jsp
in reggie/resources/
in change set [1849] included removal of unused div
tags for canvas
objects, that contained a style
setting without closing '"' character. If these div
tags should be desired in the future, the style
setting should be corrected.
comment:25 by , 12 years ago
(In [1850]) Refs #461. Java class/file ScanBQuarterMonthReportServlet.java
in reggie/src/net/sf/basedb/reggie/servlet/
updated in private method String createCaseSummaryButton(DbControl dc, String caseName)
to generate code for an HTML link, instead of a button, in order for the mouse cursor to indicate that the element could be clicked on. The "onerror
" code in the icon "img
" tag was removed, since it wasn't parsed correctly.
comment:26 by , 12 years ago
(In [1851]) References #461 and #462.
- Added print functionality to the "Sample source report" function.
- Minor adjustments to the print template to center-align the printed output on the page if possible.
- Hide the "Case summary" links on the "Samples processing statistics" report when printing.
- Force page break before the "Appended info" section since it seems like it often breaks in the middle of the table.
- Get rid of hardcoded line-breaks between images in the "Samples processing statistics" so that scaling down a printout may create a 2-column layout. Or even 3 or 4 columns, but this create very small plots.
comment:27 by , 12 years ago
comment:28 by , 12 years ago
comment:29 by , 12 years ago
(In [1854]) Refs #461. JavaScript file boxplot.js
in reggie/resources/
updated in function createBoxPlot(...)
to determine the number of decimals used for y-axis scale marker numbers. The algorithm will inspect the decimal parts of all scale marker values and select the number of decimals needed to make the largest decimal part displayed with one digit. However, if the first decimals after rounding are ".000" for all the decimal parts, no decimals will be printed.
comment:30 by , 12 years ago
comment:31 by , 12 years ago
(In [1856]) Refs #461. JavaScript file boxplot.js
in reggie/resources/
updated in function createBoxPlot(...)
in algorithm to determine the number of decimals used for y-axis scale marker numbers. The scale values are now rounded to 3 decimals before inspection of the decimal parts, to fix cases like e.g. 2.99999993, which previously would have been displayed with one decimal as "3.0", instead of "3".
comment:32 by , 12 years ago
comment:33 by , 12 years ago
(In [1858]) References #461: Sample processing report generator
Do not display seconds for RNAlater timestamps in the "appended info" table since they are only recorded with minutes. Refactored code to use DateToStringConverter
instead of methods in the ReportTableUtilServlet
(which has been removed).
comment:34 by , 12 years ago
Design comment:
When a specific chart site is selected for the SCAN-B quarter/month report, the filtering in class ScanBQuarterMonthReportServlet
is applied as follows:
- When the samples are processed, a site filter is applied when creating the lists
List<Sample> sampleTrimmedList
andList<Sample> sampleHistologyList
. - When the extracts are processed, only extracts with parent samples in
sampleTrimmedList
are placed in the listList<Extract> extractLysateList
. - When DNA and RNA extracts are processed, only extracts with parent extracts in
extractLysateList
are placed in the listsList<Extract> extractDnaList
andList<Extract> extractRnaList
, respectively. The id values of the extract and grand parent sample are placed inHashMap<Integer, Integer> extractIdSampleIdHashMap
. - When RNA QC extracts are processed, only RNA QC extracts with parents extracts used as keys in
HashMap<Integer, Integer> extractIdSampleIdHashMap
are placed in theList<Extract> extractRnaQcList
.
Since the lists sampleTrimmedList
, sampleHistologyList
, extractDnaList
, extractRnaList
, and extractRnaQcList
are the ones used, when the statistics data is calculated, all items will have a reference to a sample, resulting in only data from the selected site to be included. Future changes in the way items are selected for these and other lists used for the statistics, should take care that the site filtering works for all lists.
comment:35 by , 12 years ago
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
Ticket closed as the first version of the SCAN-B quarter/month report has been added.
comment:36 by , 12 years ago
Design update:
- The SCAN-B quarter/month report should obtain the sites for the chart site menu dynamically from server data, instead of having the data hard-coded in JSP file
scanbquartermonthreportgenerator.jsp
inreggie/resources/
.
comment:37 by , 12 years ago
(In [1997]) Refs #461. The SCAN-B quarter/month report is updated to obtain the sites for the chart site menu dynamically from server data, instead of having the data hard-coded in JSP file scanbquartermonthreportgenerator.jsp
in reggie/resources/
:
- JSP file
scanbquartermonthreportgenerator.jsp
inreggie/resources/
updated in functiongotoStep2()
to call new functiongetSites()
to obtain a JSON object with site data from the server, and construct the site-specific part of the chart site menu dynamically. New functiongetSites()
makes an Ajax call to servletScanBQuarterMonthReportServlet
to obtain the JSON site data. - Class/file
ScanBQuarterMonthReportServlet.java
inreggie/src/net/sf/basedb/reggie/servlet/
updated in protected methodvoid doGet(HttpServletRequest req, HttpServletResponse resp)
to repond to commandgetsites
by calling new private methodJSONObject createSitesList(JSONObject json)
and returning the JSON object. New private methodJSONObject createSitesList(JSONObject json)
returns a JSON object containing JSON site data.
comment:38 by , 11 years ago
(In [2008]) Refs #461. The SCAN-B quarter/month report is updated to avoid problems when no data is available for the site/quantity/time period selected for the plot:
- Class/file
ScanBQuarterMonthReportServlet.java
inreggie/src/net/sf/basedb/reggie/servlet/
updated in private methodJSONObject createJSONPlotStatistics(...)
to check if variablefloatPct25
differs fromnull
, before using the value in a conditional. - JavaScript file
reports/boxplot.js
inreggie/resources/
updated in functioncreateBoxPlot(boxPlotJsonObject, canvas, draw_area_wdt, draw_area_hgt, draw_scale_factor)
to check if an element inboxPlotJsonObject.valueGuideLinesY
differs fromnull
, before using the value.
Ticket accepted.