MeLuDI v.1.2.5 bug fixes and improvements

[olle]

This ticket covers bug fixes and improvements to MeLuDI v.1.2.5.

[olle]

Ticket accepted.

[olle]

Ticket accepted.

[olle]

Traceability note:

• MeLuDi v.1.0 was introduced in Ticket #690 (MeLuDi - Registration wizard for melanoma-lung cancer projects).
• MeLuDi v.1.1 was introduced in Ticket #718 (MeLuDi v.1.0 bug fixes and improvements).
• MeLuDi v.1.2 was introduced in Ticket #725 (MeLuDi v.1.1 bug fixes and improvements).
• MeLuDI v.1.2.1 was introduced in Ticket #759 (MeLuDi v.1.2 bug fixes and improvements).
• MeLuDI v.1.2.2 was introduced in Ticket #767 (MeLuDI v.1.2.1 bug fixes and improvements).
• MeLuDI v.1.2.3 was introduced in Ticket #768 (MeLuDI v.1.2.2 bug fixes and improvements).
• MeLuDI v.1.2.4 was introduced in Ticket #777 (MeLuDI v.1.2.3 bug fixes and improvements).
• MeLuDI v.1.2.5 was introduced in Ticket #786 (MeLuDI v.1.2.4 bug fixes and improvements).
• First version of library preparation wizards was introduced in Ticket #801 (MeLuDI library preparation wizards first version).

[olle]

(In [3379]) Refs #792. Updates due to version number change to "1.3-dev".

1. Version number change to "1.3-dev":
   a. Outermost Ant XML build file build.xml in / updated.
   b. Extensions XML configuration file extensions.xml in META-INF/ updated.
   c. Javascript file meludi-2.js in resources/ updated.
   d. Javascript file samplereportgenerator.js in resources/reports/ updated.
   e. Java class/file Meludi.java in src/net/sf/basedb/meludi/ updated.

[olle]

(In [3380]) Refs #792. Configuration file extensions.xml in META-INF updated by resetting min-base-version to 3.4.0. The value was prematurely updated to 3.5.0 in change set ​3379, but this change will be committed later together with corresponding changes in other configuration files.

[olle]

(In [3381]) Refs #792. Ant build and other configuration files updated to require minimum BASE version 3.5.0. The objective is to get access to the new ItemList item:

1. Minimum BASE version changed to 3.5.0:
   a. Class path file .classpath in / updated.
   b. Outermost Ant XML build file build.xmlin / updated.
   c. Extensions XML configuration file extensions.xml in META-INF/ updated.

[olle]

Design discussion:

At MeLuDI meeting 2015-06-08 on planning for Library Preparation wizards, some general decisions were made, that affect previous wizards:

• Support for "Extra material", i.e. samples/extracts not included in the main MeLuDI project, should be added to all stages of the MeLuDI pipeline. This includes the case registration wizards, but details regarding name conventions and sample types etc. have not been decided yet. The new policy means that values for different quantities should be stored for extra material, as well as for normal MeLuDI samples/extracts.
• Currently, extract source item selections in DNA/RNA extraction wizards are stored using annotations for a start plate; a "DNA reaction plate" library plate with name consisting of "LP" plus a 5-figure number, on which the DNA extracts were intended to be placed. However, placement on the start plate is first made during library preparation, and it is not certain that DNA extracts for all relevant items coupled to the plate through annotations, would finally be placed on the plate. Also, a start plate might include DNA extracts for source items coupled to another start plate. Therefore, it seemed natural to not introduce any start plate until library preparation, when DNA extracts are placed on it. A consequence is that extract source item selections in DNA/RNA extraction wizards must be stored in another item than a start plate.

Design considerations:

1. A plausible candidate item for storing an item selection is the BASE item type "ItemList". However, this item type was introduced first in BASE v3.5.0, while MeLuDI is currently used with BASE v3.4.*. Item lists were introduced in BASE Ticket ​#1325 "Lists of items (similar funcion as biomaterial lists)". Reggie ticket #750 (Changes required for BASE version 3.5) describes what changes are necessary for a BASE extension, and the update of MeLuDI for use with BASE 3.5.* will be based on this (although MeLuDI requires less changes than Reggie).

[olle]

(In [3384]) Refs #792. MeLuDI updated to be able to be used with BASE version 3.5. Fixes consist of replacing BioMaterialList with ItemList, and Item.BIOMATERIALLIST with Item.ITEMLIST:

1. Data access object file BiomaterialList.java in src/net/sf/basedb/meludi/dao/ updated.
2. Java servlet classes files ExtractionServlet.java (removed import only) and InstallServlet.java in src/net/sf/basedb/meludi/servlet/ updated.

[olle]

(In [3385]) Refs #792. MeLuDI updated to be able to be used with BASE version 3.5. Fixes use of reprecated methods:

1. Data access object files Patient.java and ReactionPlate.java in src/net/sf/basedb/meludi/dao/ updated.

[olle]

(In [3408]) Refs #792. One new site, "Trelleborg", added, belonging to RCC-syd:

1. Java class/file Site.java in src/net/sf/basedb/meludi/ updated by adding entry for "Trelleborg".
2. Java servlet class/file InstallServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doGet(HttpServletRequest req, HttpServletResponse resp) by adding new site entry "Trelleborg" as allowed value for Annotationtype.SITE.
3. JSP file persinfo.jsp in resources/personal/ updated by adding new site entry "Trelleborg" to site select options.
4. JSP file specimentube.jsp in resources/sampleproc/ updated by adding new site entry "Trelleborg" to site select options.

[olle]

Design discussion regarding extra material:

At MeLuDI meeting 2015-06-15, the following details regarding name conventions etc. for extra material were decided:

1. The following data is expected to be known for an extra material:
   a. Input name.
   b. Type. One of Specimen, DNA, or RNA.
   c. Arrival date.
   d. Operator delivery comment (optional).
2. The original (input) name of an extra material sample/extract does not have to follow any strict name convention, except:
   a. The name must not start with letters "ML", since this is reserved for samples/extracts in the MeLuDI project.
   b. The input name may consist of English letters, numbers, hyphens "-" and underscores "_". It must contain at least one letter.
3. No case item will be defined for an extra material item, but the input name will have the same function as the case name for an item in the MeLuDI project. The program will add a suffix to the input name of the extra material. The suffix for a specimen will be a dot "." plus a number, starting with "1"; for a DNA item it will be ".d"; and for an RNA item it will be ".r".
4. Only one DNA and one RNA item is allowed for an input name of extra material. In addition, a number of specimens are allowed for an input name, even if a DNA and/or RNA item is registered for it. (The latter is not allowed for items in the MeLuDI project.)
5. No tracking of plate positions will be performed for extra material items.
6. For each extra material specimen registered, a DNA and an RNA extract for the specimen should be registered, but without plate positions.

[olle]

Functional specification update:

• The case registration section should be extended with a wizard for registering extra material.

The wizard should contain two steps:

1. The following data is entered:
   a. The number of extra material to be added.
   b. The default arrival date (default is current date).
2. The wizard contains of one line for each extra material, containing the following columns:
   a. Name input field.
   b. Type selection by radio button for DNA/RNA/Specimen ("DNA" is default).
   c. Item name with type suffix (added by program).
   d. Arrival date input field (default is date entered in step 1).
   e. Operator delivery comment input area.
   f. Status icon (added by program).
   g. Input message (added by program).

Checks that should be performed on input data:

1. Normal consistency checks on number and date entries.
2. Check that arrival date is not a future date.
3. Check that input name is not empty, and that it does not start with "ML".
4. For a DNA or RNA entry, it should be checked that no extract with input name and type already exists in the database.
5. For specimen entries with a given input name, the number of existing specimens with this name should be checked. The suffices for the new specimen entries should start by adding one to the highest suffix number of existing specimen items with the given name. As an example, if two new specimens with input names "Test_Item_0A" should be added, and specimens "Test_Item_0A.1" and "Test_Item_0A.2" already exist, the new specimens should get item names "Test_Item_0A.3" and "Test_Item_0A.4".

Design update:

1. JSP file index.jsp in resources/ should be updated in the case registration section by adding a link to a wizard for registering new extra material. The link should be to new JSP file extramaterial.jsp in resources/sampleproc/.
2. New JSP file extramaterial.jsp in resources/sampleproc/ added. It contains two steps and is coupled to new javascript file extramaterial.js in resources/sampleproc/.
3. New javascript file extramaterial.js in resources/sampleproc/ added. It contains the following functions (among other):
   a. Function initPage() adds event and button click handlers, enters today's date as default arrival date, and displays step one of the wizard.
   b. Function validateStep1(event) calls function nofExtraMaterialOnChange() to check that the number of extra materials is OK.
   c. Function initializeStep2() creates a table with one input row for each new extra material to be added, and adds event handlers for type radio buttons and arrival date fields.
   d. Function nameOnChange() performs consistency checks on the input name, and calls servlet SpecimenTubeServlet with command "GetExtraMaterialInfo" and callback function extraMaterialInfoLoaded(response) to perform checks requiring the database and create an item name.
   e. Function extraMaterialInfoLoaded(response) performs checks requiring the database and creates an item name, if no errors are found. For DNA and RNA items are checked that no previous item with same name and type exists. Suffix numbers for new specimen items are obtained by adding one to the highest suffix number of existing specimen items with the given name. For a new specimen, function updateSpecimenItemNames(currentName, highestSpecimenNumber) is called to update item names.
   f. Function updateSpecimenItemNames(currentName, highestSpecimenNumber) updates all item names for specimens with given name by adding numbers above those already used for specimens in the database.
   g. Function typeOnChange(event) calls function nameOnChange(event), since a change of type implies a change in suffix for item name.
   h. Function arrivalDateOnChange(event) performs date consistency checks for arrival date entries.
   i. Function validateStep2(event) performs basic validation checks (currently only that the number of extra material is valid).
   j. Function submit() stores the input data in a JSON array and submits it to servlet SpecimenTubeServlet with command "CreateExtraMaterial" and callback function submissionResults(response) using a "POST" request.
   k. Function submissionResults(response) displays the response message after registration.
4. Java servlet class/file SpecimenTubeServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated with new command "GetExtraMaterialInfo". It obtains information of the number of specimen, DNA, and RNA items with a given input name, as well as the highest suffix number for specimens with that input name.
   b. Protected method void doPost(HttpServletRequest req, HttpServletResponse resp) updated with new command "CreateExtraMaterial". It creates new extra material items, and for specimen items in the database, related DNA and RNA items.

[olle]

(In [3412]) Refs #792. MeLuDI updated by adding registration of extra material:

1. JSP file index.jsp in resources/ updated in the case registration section by adding a link to a wizard for registering new extra material. The link should be to new JSP file extramaterial.jsp in resources/sampleproc/.
2. New JSP file extramaterial.jsp in resources/sampleproc/ added. It contains two steps and is coupled to new javascript file extramaterial.js in resources/sampleproc/.
3. New javascript file extramaterial.js in resources/sampleproc/ added. It contains the following functions (among other):
   a. Function initPage() adds event and button click handlers, enters today's date as default arrival date, and displays step one of the wizard.
   b. Function validateStep1(event) calls function nofExtraMaterialOnChange() to check that the number of extra materials is OK.
   c. Function initializeStep2() creates a table with one input row for each new extra material to be added, and adds event handlers for type radio buttons and arrival date fields.
   d. Function nameOnChange() performs consistency checks on the input name, and calls servlet SpecimenTubeServlet with command "GetExtraMaterialInfo" and callback function extraMaterialInfoLoaded(response) to perform checks requiring the database and create an item name.
   e. Function extraMaterialInfoLoaded(response) performs checks requiring the database and creates an item name, if no errors are found. For DNA and RNA items are checked that no previous item with same name and type exists. Suffix numbers for new specimen items are obtained by adding one to the highest suffix number of existing specimen items with the given name. For a new specimen, function updateSpecimenItemNames(currentName, highestSpecimenNumber) is called to update item names.
   f. Function updateSpecimenItemNames(currentName, highestSpecimenNumber) updates all item names for specimens with given name by adding numbers above those already used for specimens in the database.
   g. Function typeOnChange(event) calls function nameOnChange(event), since a change of type implies a change in suffix for item name.
   h. Function arrivalDateOnChange(event) performs date consistency checks for arrival date entries.
   i. Function validateStep2(event) performs basic validation checks (currently only that the number of extra material is valid).
   j. Function submit() stores the input data in a JSON array and submits it to servlet SpecimenTubeServlet with command "CreateExtraMaterial" and callback function submissionResults(response) using a "POST" request.
   k. Function submissionResults(response) displays the response message after registration.
4. Java servlet class/file SpecimenTubeServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated with new command "GetExtraMaterialInfo". It obtains information of the number of specimen, DNA, and RNA items with a given input name, as well as the highest suffix number for specimens with that input name.
   b. Protected method void doPost(HttpServletRequest req, HttpServletResponse resp) updated with new command "CreateExtraMaterial". It creates new extra material items, and for specimen items in the database, related DNA and RNA items.

[olle]

Functional specification update:

• The Lab tracking protocol for Allprep isolation wizard should be updated to support registered extra material.

Design considerations:

In many cases this implies a simplification, since extra material now no longer represents a special case. One exception is the order in which items are listed in selection lists, tables, and protocols. It was decided that it is preferable to first list MeLuDI items, then extra material items.

The full update will be split up into several smaller parts. The first update fixes the selection and presentation steps of the wizard, while registration and support functions like creation of NanoDrop and qPCR sample ID files will be fixed later.

[olle]

(In [3413]) Refs #792. Lab tracking protocol for Allprep isolation wizard updated in selection lists and presentation tables and protocols to list registered extra material items after MeLuDI items:

1. JSP file extraction_preparation.jsp in resources/sampleproc/ updated by removing code related to number of extra material items, since these are now included with the other items.
2. Javascript file extraction_preparation.js in resources/sampleproc/ updated by removing code related to creating list of extra material items, since these are now included with the other items.
3. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated for command "GetUnprocessedItems" to list MeLuDI items first, then extra material items, each sorted alphabetically in its group.

[olle]

(In [3414]) Refs #792. Lab tracking protocol for Allprep isolation wizard updated in convenience method identifying specimen items from name to no longer exclude extra material items. This will allow creation of NanoDrop and qPCR sample ID files to include extra material items after MeLuDI items:

1. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Private method boolean itemIsSpecimen(String itemName) updated to no longer require that a specimen item name starts with "ML".

[olle]

(In [3416]) Refs #792. Java servlet classes/files updated by referring to a single definition of item name prefix for MeLuDI items, placed in java class/file Meludi.java in src/net/sf/basedb/meludi/:

1. Java class/file Meludi.java in src/net/sf/basedb/meludi/ updated with new public static final String ITEM_NAME_PREFIX, set to "ML".
2. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated for command "GetUnprocessedItems" to refer to Meludi.ITEM_NAME_PREFIX.
3. Java servlet class/file LabelGenerationServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated for command "GetLabelFile" to refer to Meludi.ITEM_NAME_PREFIX.

[olle]

(In [3418]) Refs #792. Data access object class/file SpecimenTube.java in src/net/sf/basedb/meludi/dao/ updated with new public static method List<SpecimenTube> findByBaseName(DbControl dc, String name). It finds all specimen tubes that are linked by base name (name before first dot ".") to a given name. This change should have been included in change set ​[3412], since SpecimenTubeServlet calls this method for command "GetExtraMaterialInfo", that was introduced in that change set.

[olle]

(In [3423]) Refs #792. In order for case summary to work for extra material items without a case, where the extra material item name is used instead of case name, suffix removal in different data access object methods should not rely on the case name to be of a specific length:

1. Data access object class/file Dna.java in src/net/sf/basedb/meludi/dao/ updated in public static method List<Dna> findByCaseName(DbControl dc, String name) to remove suffices by checking for the first dot ".".
2. Data access object class/file Rna.java in src/net/sf/basedb/meludi/dao/ updated in public static method List<Rna> findByCaseName(DbControl dc, String name) to remove suffices by checking for the first dot ".".
3. Data access object class/file SpecimenTube.java in src/net/sf/basedb/meludi/dao/ updated in public static methods List<SpecimenTube> findByCase(DbControl dc, Case theCase, String originalCaseName) and List<SpecimenTube> findByCaseName(DbControl dc, String name) to remove suffices by checking for the first dot ".".

[olle]

Design discussion for updating the DNA/RNA extraction wizards to not include any library plate:

An ItemList item, introduced in BASE v3.5.0, has previously been suggested as a candidate for storing the data currently stored in annotations for a start plate. However, a single ItemList can only contain member items of identical types, such as samples and extracts. Since extract source items include specimens (samples) and DNA/RNA (extracts), it is not possible to store a general selection of extract source items in a single item list. Some solutions are:

1. Storing all data in annotations, as was done with start plates. This means that one does not use the ability of an item list to contain other items, but just uses it as a "dummy" item to add annotations to.
   Pro. Only minor modifications of the code are needed.
   Con: When inspecting an item list in BASE, no links to the extract source items exist, only their names.
2. Storing extract source items in a pair of item lists, one for storing sample items, and one for storing extract items. The fact that two item lists are used for each selection should be hidden from the user. The item lists in a pair could be given names only differing by a suffix, e.g. ".s" for the one storing samples and ".e" for the one storing extracts, where the common basename without suffices is the one presented in selection lists.
   Pro: When inspecting an item list in BASE, links to extract source items exist, although in two different item lists for a single selection.
   Con: Requires more modification of the code.
3. Storing items in a single item list for extracts. A specimen is represented by its DNA child item, while input DNA and RNA items are stored directly. The fact that DNA child items are used to represent specimens should be hidden from the user.
   Pro: When inspecting an item list in BASE, links to extract source items or their DNA child items exist. For specimens this means that one has to click at least twice to get to the desired specimen item overview.
   Con: Requires more modification of the code.

For all solutions, data other than extract source items will be stored as annotations, just as for start plates. For technical reasons, the annotations cannot have exactly the same name as used for a start plate, but use of a common prefix can make the connection clear (the user should not have to know the internal name of the annotation, anyway).

After some consideration. solution 3 was chosen for updating the DNA/RNA extraction wizards. The downside of representing a specimen by its DNA child item, was compensated by having a single item list for each selection, links to items or its children in the BASE view, and the fact that the number of member items in an item list equals the number of selected extract source items.

[olle]

(In [3430]) Refs #792. MeLuDI updated by adding new ItemList annotations for storing sample prep data previously stored as annotations for a start DNA reaction plate. Also new annotation for keeping track on whether an item list has been processed or not:

1. Data access object class/file Annotationtype.java in src/net/sf/basedb/meludi/dao/ updated with new annotation types for item lists.
2. Java servlet class/file InstallServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doGet(HttpServletRequest req, HttpServletResponse resp) to include the new extract annotation types.

[olle]

(In [3431]) Refs #792. MeLuDI updated by preparing for use of ItemList objects to store extract source item selections, previously managed by start DNA reaction plates:

1. Utility class/file JsonUtil.java in src/net/sf/basedb/meludi/ updated by adding new public static method JSONObject getItemListAsJson(ItemList l). Value of the "size" JSON key is set to the number of member items in the item list.
2. Utility class/file Meludi.java in src/net/sf/basedb/meludi/ updated with new static final String START_LIST_NAME_PREFIX.
3. New data access object class/file StartList.java in src/net/sf/basedb/meludi/dao/ added. It contains a constructor for creating a StartList item from an ItemList, and public static methods for finding general and active start lists by name prefix.

[olle]

(In [3432]) Refs #792. MeLuDI updated in main sample preparation wizards to use item lists to store extract source item selections and related data, instead of start DNA reaction plates:

1. JSP file index.jsp in resources/ updated to refer to new item list counters for sample preparation entries.
2. Most JSP and javascript files in resources/sampleproc/ updated to use item lists to store extract source item selections and related data, instead of start DNA reaction plates. The exception is JSP and javascript files for pop-up dialogs used to edit some sample preparation data, since these are not invloved in storing or retrieving data from the database.
3. Java class/file CounterService.java in src/net/sf/basedb/meludi/counter/ updated with counters for general and unprocessed start lists.
4. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated with new commands "GetStartItemLists", "GetUnprocessedStartItemLists", "GetSourceItemListFromStartItemList", and "GetNextStartListName". The name of commend "GetSourceItemListFromStartItemList" was chosen to be consistent with previous commands for getting source items, but unfortunately becomes ambiguous when ItemList items were introduced; it should be interpreted as "Get list of source items from start ItemList".
   b. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated for commands "DownloadNanoDropSampleFile", "DownloadQPcrSampleIdFile", and "DownloadReagentLotNoFile" to use a start list name instead of start plate name as input.
   c. Protected method void doPost(HttpServletRequest req, HttpServletResponse resp) updated with new command "RegisterStartItemList".
   d. Protected method void doPost(HttpServletRequest req, HttpServletResponse resp) updated for command "RegisterDnaRna" to use a start item list instead of start DNA reaction plate. Also updated to make the program more stable when trying to retrieve float values from non-existing variables.
   e. New private convenience methods boolean itemIsMeludiItem(String itemName), ItemList findItemListWithHighestNumber(DbControl dc, String itemListPrefix), List<String> fetchExtractSourceNameListFromItemList(DbControl dc, ItemList startList), and List<String> fetchExtractSourceNameListFromItemList(DbControl dc, ItemList startList, boolean meludiItemsFirst).

[olle]

(In [3433]) Refs #792. MeLuDI sample preparation code refactored:

1. Utility class/file Meludi.java in src/net/sf/basedb/meludi/ updated with five new public static methods, and one private static convenience method:
   a. boolean itemIsSpecimen(String itemName).
   b. boolean itemIsRna(String itemName).
   c. boolean itemIsDna(String itemName).
   d. boolean itemIsExtractSourceItem(String itemName).
   e. boolean itemIsMeludiItem(String itemName).
   f. Private static convenience method int numberOfSpecificCharacters(String str, char c) added.
2. Data access object class/file StartList.java in src/net/sf/basedb/meludi/dao/ updated with new public methods:
   a. List<String> fetchExtractSourceNameListFromItemList(DbControl dc).
   b. List<String> fetchExtractSourceNameListFromItemList(DbControl dc, boolean meludiItemsFirst).
3. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated to use new public methods in classes Meludi and StartList, instead of own private methods.

[olle]

(In [3434]) Refs #792. MeLuDI updated in case summary to report start lists instead of start DNA reaction plates for items:

1. JSP and javascript files case_summary.jsp and case_summary.js in resources/reports/ updated to report start lists instead of start DNA reaction plates for items.
2. Java servlet class/file CaseSummaryServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated for command "GetCaseInfo" to report start lists instead of start DNA reaction plates for items.
   b. New private method ItemList fetchStartList(DbControl dc, SpecimenTube st) added. It calls new private method ItemList fetchStartList(DbControl dc, String name) to get the start list from the specimen tube sample name.
   c. New private method ItemList fetchStartList(DbControl dc, String name) added. It finds the first start item list with the input item name represented in the member item list, or, in case of a specimen, as a parent item to one of the member items.

[olle]

Functional specification update:

• The registration wizard for extra material should be updated with additional checks:
  a. For DNA/RNA extract entries, it is currently checked that no item with the same name exists in the database. In addition, it should be checked that only one extract item of a specific type (DNA/RNA) and name exists in the input list. If more than one such entry exists in the input list, the first should be considered OK, while any other should be marked as invalid due to being a duplicate.
  b. At start, all extra material entries in the list should be marked as invalid due to being missing.
  c. Registration should be prohibited as long as any input entry is marked as invalid.

Design discussion:

All the changes above can be made in javascript file extramaterial.js in resources/sampleproc/. The desired duplicate check for extract items is complicated by the fact that change of an item's name or type, might not only affect the validity of the item itself, but also other items below it in the input list. Of the other items, both those with the old name and the new name might be affected; extract items in validity status, and specimen items in number suffix. The following calling chain of event handlers is proposed as a solution to this problem:

Function	Trigger event	Checks/Function calls
typeOnChange(event)	Type change (DNA/RNA/Spec.)	Calls nameOnChange(event)
nameOnChange(event)	Name change, function call	Checks name consistency. If OK, Ajax call to SpecimenTubeServlet with command "GetExtraMaterialInfo" and callback function extraMaterialInfoLoaded(response)
extraMaterialInfoLoaded(response)	(Callback function)	a. Checks for duplicate extracts in database. b. Calls function checkExtractItemNames() to check validity of all entries. c. Sets valiidty of all extracts items, and number suffices for specimens.
checkExtractItemNames()	Function call	Checks for duplicate extract names in list for all items. Result returned as JSONArray.

Design update:

Javascript file extramaterial.js in resources/sampleproc/ updated:

1. Function initializeStep2() updated to set initial input status for all items to invalid due to missing name.
2. New global variable ExtraMaterialIsValid added, initialized to false. It is set to true at start of function nameOnChange(event), and is then reset to false if input status for any entry is set to invalid in this function or extraMaterialInfoLoaded(response). Function validateStep2(event) checks its value, when determining if registration is possible.
3. Callback function extraMaterialInfoLoaded(response) updated:
   a. New function checkExtractItemNames() is called to perform a check for duplicate extract names in input list. The result is used to mark all duplicate entries as invalid.
   b. Value of new global variable ExtraMaterialIsValid is set to false if input status for any entry is set to invalid.
4. New function checkExtractItemNames() added. It performs a check for duplicate names of extracts in input list. The first occurrence is considered OK, while following are considered duplicates. Results are returned in a JSONArray, with value false for duplicates, else true.

[olle]

(In [3435]) Refs #792. Registration wizard for extra material updated to check for duplicate extract entries in input list. Also initializing input status for all entries as invalid due to missing name, and preventing registration, if input status for any entry is invalid.

Javascript file extramaterial.js in resources/sampleproc/ updated:

1. Function initializeStep2() updated to set initial input status for all items to invalid due to missing name.
2. New global variable ExtraMaterialIsValid added, initialized to false. It is set to true at start of function nameOnChange(event), and is then reset to false if input status for any entry is set to invalid in this function or extraMaterialInfoLoaded(response). Function validateStep2(event) checks its value, when determining if registration is possible.
3. Callback function extraMaterialInfoLoaded(response) updated:
   a. New function checkExtractItemNames() is called to perform a check for duplicate extract names in input list. The result is used to mark all duplicate entries as invalid.
   b. Value of new global variable ExtraMaterialIsValid is set to false if input status for any entry is set to invalid.
4. New function checkExtractItemNames() added. It performs a check for duplicate names of extracts in input list. The first occurrence is considered OK, while following are considered duplicates. Results are returned in a JSONArray, with value false for duplicates, else true.

[olle]

(In [3436]) Refs #792. Lab tracking (protocol/)report for Allprep isolation updated in data section between header and item table to include lot numbers for Proteinase K and DNAse mix:

1. JSP file extraction_protocol2.jsp in resources/sampleproc/ updated with tags for Proteinase K and DNAse mix lot numbers.
2. Javascript file extraction_protocol.js in resources/sampleproc/ updated in function initializeProtocol(submitInfoJsonStr) to retrieve values for Proteinase K and DNAse mix lot numbers from input JSON data and insert them in the created protocol.
3. Javascript file extraction_registration.js in resources/sampleproc/ updated in function createProtocol() to add Proteinase K and DNAse mix lot numbers to submitted JSON data.
4. Javascript file extraction_formcopy.js in resources/sampleproc/ updated in function createProtocol() to add Proteinase K and DNAse mix lot numbers to submitted JSON data.

[olle]

Functional specification update:

• The Lab tracking (protocol/)report for Allprep isolation and the wizards creating them should be updated. The first three updates concerns data in the section between header and item table.

1. Lot numbers for Proteinase K and DNAse mix should be included in the data section (the numbers are already available). This was included in change set ​3436.
2. The DNA/RNA registration/quantification wizard should be updated to allow change of the FFPE DNA/RNA kit lot number, originally input in the Lab tracking protocol for Allprep isolation wizard. Currently the value is displayed in the former wizard, but the input field is disabled.
3. The DNA/RNA registration/quantification wizard should be updated to allow input of a Quality Control Template (QCT) lot number, and the value should be included in the report data section.
4. The DNA/RNA registration/quantification wizard should be updated to set any negative concentration values to 0.

Design discussion:

• Regarding request (b) above: Allowing change of the FFPE DNA/RNA kit lot number in the DNA/RNA registration/quantification wizard: Since this value is stored as annotation Annotationtype.LIST_ALLPREP_FFPE_KIT in the start item list, the latter should be updated with the new value, if it is changed in the DNA/RNA registration/quantification wizard.

[olle]

(In [3437]) Refs #792. The DNA/RNA registration/quantification wizard updated to allow change of the FFPE DNA/RNA kit lot number:

1. Javascript file extraction_registration.js in resources/sampleproc/ updated:
   a. Function sourceItemListLoaded(response) updated to not disable input field for FFPE DNA/RNA kit lot number.
2. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated:
   a. Protected method void doPost(HttpServletRequest req, HttpServletResponse resp) updated for command "RegisterDnaRna" to update the value of start item list annotation Annotationtype.LIST_ALLPREP_FFPE_KIT with value from input field in form.

[olle]

(In [3438]) Refs #792. The DNA/RNA registration/quantification wizard updated to allow input of a Quality Control Template (QCT) lot number. The value is included in the (protocol)/report data section.

1. JSP and javascript files in resources/sampleproc/ updated to allow input of a Quality Control Template (QCT) lot number, which is included in the (protocol)/report data section. The code additions are based on that for Quality Control Primers (QCP) RGT number:
   a. edit_dnarna_details.jsp
   b. extraction_formcopy.js
   c. extraction_formcopy.jsp
   d. extraction_preparation.js
   e. extraction_protocol.js
   f. extraction_protocol2.jsp
   g. extraction_registration.js
   h. extraction_registration.jsp
2. Data access object class/file Annotationtype.java in src/net/sf/basedb/meludi/dao/ updated with new extract annotation type Annotationtype.QUALITY_CONTROL_TEMPLATE_QCT.
3. Java servlet class/file InstallServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doGet(HttpServletRequest req, HttpServletResponse resp) to include the new extract annotation type.
4. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated to support a Quality Control Template (QCT) lot number. The code additions are based on that for Quality Control Primers (QCP) RGT number.

[olle]

(In [3439]) Refs #792. DNA/RNA registration/quantification wizard updated to set any negative concentration values to 0:

1. Javascript file extraction_registration.js in resources/sampleproc/ updated in functions rnaConcOnChange(event) and dnaConcOnChange(event) to set any input negative concentration value to 0.

[olle]

(In [3440]) Refs #792. MeLuDI has been updated in parts of the existing code, as a preparation for the first library preparation wizards:

1. Java convenience class/file Meludi.java in src/net/sf/basedb/meludi/ has been updated with javadoc comments on methods, used to identify certain types of items from their names. Some of the algorithms used may work at present, but not when more item types are added. Accurate javadoc documentation should hopefully inform, if a method is appropriate or not in future situations.
2. Data access object class/file MeludiItem.java in src/net/sf/basedb/meludi/dao/ has been updated to allow a snapshot manager to be specified in methods fetching annotation values.
3. Data access object class/file StartList.java in src/net/sf/basedb/meludi/dao/ has been updated with new methods for obtaining names of items of a specific type from an item list.
4. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ has been updated by using snapshot manager, when obtaining annotations, refactoring commands to get start plates by calling a new private convenience method JSONArray fetchStartItemLists(DbControl dc, Boolean processed, Boolean orderDesc), and adding new command for finding items from members of start item lists.

[olle]

Functional specification update:

• In order to facilitate the change from storing item data for the DNA/RNA extraction step in annotations for a start DNA plate to a start item list, a wizard creating start lists from start plates should be added to the /resources/admin/ section. This wizard is only intended to be used once, when upgrading from version v1.2.* to v1.3, and might therefore be removed in future versions.

The following assumptions must be met for success:

1. The start plate contains a list of all related MeLuDI item names as annotation.
2. The start plate contains a list of all related extra material names as annotation.
3. All related specimens and extracts have already been created with the specified names.

For each start plate, a start item list will be created, with the following properties:

1. The start list name is that of the start plate with prefix "LP" exchanged for "SL".
2. All relevant start plate annotations are copied to corresponding start list annotations.
3. Items named in start plate annotations are added as start list members (DNA child extracts represent specimens).
4. An optional start plate creation date is stored in start list "ListExtractionDate" annotation.

Storage of the start plate creation date in an item list annotation was chosen, as creation events are not defined for item lists. Since a set creation date for a start plate in MeLuDI v1.2.* indicates that the plate has been processed, i.e. that concentration and delta-Ct values have been registered for MeLuDI DNA/RNA, it might be argued that annotation "SamplePrepListIsProcessed" for the created start list should be set to true, in order t0 indicate that the start list has been processed. However, this is normally only true for converted item lists without extra material, since concentration and delta-Ct values were only registered for MeLuDI items in v1.2.*, but not for extra material. Setting the value of annotation "SamplePrepListIsProcessed" to true, would have prevented using the DNA/RNA registration/quantification wizard to register the latter values for the start list. Previously registered values for MeLuDI extracts will be displayed in the input form as a practical side effect, so these values need not be re-entered. Unfortunately, the latter is not true for kit lot numbers. The following procedure is therefore recommended to register additional values for an item list:

i. Open the item list in the "Lab tracking protocol/report copy" wizard.
ii. In step 2, after existing kit lot number values have been loaded, click button to download values to a reagent lot no. file. Cancel this wizard when ready.
iii. Open the item list in the "DNA/RNA registration/quantification" wizard.
iv. In step 2, select to import kit lot numbers from the file saved in (ii).
v. In step 3, enter the missing concentration and delta-Ct values for extra material, and complete the registration.

After controlling the created start lists, the start plates may be deleted. The latter will reappear in the library preparation wizards, when they are to be filled with samples (though not necessarily the same samples as were previously related to it via annotations).

Design update:

1. JSP file index.jsp in resources/ updated in "admin" section with new entry "Create start lists from start DNA plates", with link to JSP file create_start_list_from_start_plate.jsp in resources/admin/.
2. New JSP file create_start_list_from_start_plate.jsp in resources/admin/ added. It contains a single step and is coupled to new javascript file create_start_list_from_start_plate.js in resources/admin/.
3. New javascript file create_start_list_from_start_plate.js in resources/admin/ added. It presents a list of all available start DNA plates, from which any number may be selected. When the "Register" button is pressed, java servlet ExtractionServlet will be called with new command "CreateStartListsFromStartPlates" to create start item lists for the selected start DNA plates.
4. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doPost(HttpServletRequest req, HttpServletResponse resp) with new command "CreateStartListsFromStartPlates" for creating start item lists for the selected start DNA plates.

[olle]

(In [3441]) Refs #792. MeLuDI "admin" section updated with new entry for facilitating the change from storing item data for the DNA/RNA extraction step in annotations for a start DNA plate to a start item list. A new wizard will create start lists from selected start plates. This wizard is only intended to be used once, when upgrading from version v1.2.* to v1.3, and might therefore be removed in future versions.

1. JSP file index.jsp in resources/ updated in "admin" section with new entry "Create start lists from start DNA plates", with link to JSP file create_start_list_from_start_plate.jsp in resources/admin/.
2. New JSP file create_start_list_from_start_plate.jsp in resources/admin/ added. It contains a single step and is coupled to new javascript file create_start_list_from_start_plate.js in resources/admin/.
3. New javascript file create_start_list_from_start_plate.js in resources/admin/ added. It presents a list of all available start DNA plates, from which any number may be selected. When the "Register" button is pressed, java servlet ExtractionServlet will be called with new command "CreateStartListsFromStartPlates" to create start item lists for the selected start DNA plates.
4. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doPost(HttpServletRequest req, HttpServletResponse resp) with new command "CreateStartListsFromStartPlates" for creating start item lists for the selected start DNA plates.

[olle]

(In [3442]) Refs #792. Lab tracking protocol/report copy wizard updated in report section to calculate QIAcube position from item position in form, if no QIAcube position can be found as annotation for an extract. This might happen when a report is produced for a start item list created from a start DNA plate, as extract data was not registered for extra material, in MeLuDI versions prior to v1.3:

1. Javascript file extraction_formcopy.js in resources/sampleproc/ updated in function initializeStep3b(response) to calculate QIAcube position from item position in form, if no QIAcube position can be found as annotation for an extract.

[olle]

(In [3443]) Refs #792. MeLuDI "admin" section updated by placing entry for creating start lists from selected start plates in new sub-section "Special conversion tools":

1. JSP file index.jsp in resources/ updated in "admin" section by placing entry for creating start lists from selected start plates in new sub-section "Special conversion tools".
2. New image wrench.png in resources/images/ added. It is intended to be used to indicate various tool sections.

[olle]

(In [3444]) Refs #792. MeLuDI counter service updated to support counting processed start item lists:

1. Java service class/file CounterService.java in src/net/sf/basedb/meludi/counter/ updated in private method void countItemLists(DbControl dc, JSONObject json) by addition of a counter for processed start item lists, coupled to JSON key "start-lists-processed".

[olle]

(In [3445]) Refs #792. MeLuDI updated in wizard for creating start item lists from start DNA plates to mark a created start item list as processed, if the source start plate is so marked, and no extra material is related to the start list:

1. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doPost(HttpServletRequest req, HttpServletResponse resp) for command "CreateStartListsFromStartPlates" to mark a created start item list as processed if the source start plate is so marked, and no extra material is related to the start list. The start item list is indicated as processed by setting value of Boolean annotation Annotationtype.SAMPLE_PREP_LIST_IS_PROCESSED to true.

[olle]

(In [3446]) Refs #792. MeLuDI updated to allow a maximum delta-Ct value filter when obtaining DNA items from start item lists:

1. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doGet(HttpServletRequest req, HttpServletResponse resp) for command "GetListOfItemsFromStartItemLists" to accept an input Float parameter "maxDeltaCt", and if the latter is not null, only include DNA items with values of delta-Ct < the given max delta-Ct value.

[olle]

(In [3447]) Refs #792. MeLuDI updated to allow a check on whether a start plate with a given name already exists:

1. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doGet(HttpServletRequest req, HttpServletResponse resp) with new command "CheckIfStartPlateNameUsed" to check if a start plate exists with a given name. The result is returned as a JSON object with the name and optional id of the found start plate.

[olle]

A special ticket has been created to collect code changes related to the first version of library preparation wizards, Ticket #801 (MeLuDI library preparation wizards first version). Code changes in this category to MeLuDI v1.3 will recorded both there and in this ticket.

[olle]

(In [3448]) Refs #792. MeLuDI updated by removing filter for maximum delta-Ct value from Extraction servlet, since this filter is now completely implemented in javascript:

1. Java servlet class/file ExtractionServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doGet(HttpServletRequest req, HttpServletResponse resp) for command "GetListOfItemsFromStartItemLists" by removing filter for maximum delta-Ct value from Extraction servlet, since this filter is now completely implemented in javascript. Removal of code for the unused filter will hopefully increase readability of the code.

[olle]

(In [3449]) Refs #801. Refs #792. First experimental version of library preparation wizards for MeLuDI:

Updates of previous files:

1. XML servlet configuration file servlets.xml in META-INF/ updated with entries for new servlet classes LibPrepServlet and DnaServlet.
2. JSP file index.jsp in resources/ updated with new section for library preparation wizards, containing links to three wizards. In view of the experimental character of the latter (especially the ones related to protocol creation and barcode assignment), the section header has been given the extra caution "(Experimental - For testing only)".
3. CSS file plate.css in resources\css\ updated by increasing the well width from 90 to 100 pixels, in view of the longer storage location names in MeLuDI, relative to Reggie.
4. Data access object class/file Annotationtype.java in src/net/sf/basedb/meludi/dao/ updated with new annotation types related to library preparation.
5. Data access object class/file BioPlateType.java in src/net/sf/basedb/meludi/dao/ updated with new bioplate type LIBRARY, and allowing plate name prefix and number of digits in name to be editable.
6. Data access object class/file Dna.java in src/net/sf/basedb/meludi/dao/ updated with new public static method List<Dna> toList(Collection<Extract> extracts).
7. Data access object class/file MeludiRole.java in src/net/sf/basedb/meludi/dao/ updated with new roles LIBRARY_PREP and LIBRARY_PLATE_DESIGNER.
8. Data access object class/file Subtype.java in src/net/sf/basedb/meludi/dao/ updated by adding new attribute String itemSuffix, and new public method BasicItem getLatestProjectDefault(DbControl dc).
9. Java servlet class/file InstallServlet.java in src/net/sf/basedb/meludi/servlet/ updated in protected method void doGet(HttpServletRequest req, HttpServletResponse resp) to include the new extract annotation types, and install barcodes.

New files related to library preparation:

10. CSS file startplate.css in resources\css\ added. It is a modified version of plate.css, that restricts highlights to the left half of the plate, since any change in that half should be replicated with a corresponding change in the right half.
11. Image file manual_dna.png in resources/images/ added.
12. JSP and javascript files related to the first version of library preparation wizards added to resources/libprep/:
    a. assign_barcodes.js
    b. assign_barcodes.jsp
    c. auto_select_dna.js
    d. auto_select_dna.jsp
    e. dna_protocol.js
    f. dna_protocol.jsp
    g. dna_protocol2.jsp
    h. libprep_protocol.js
    i. libprep_protocol.jsp
    j. libprep_protocol2.jsp
    k. pools.js
    l. select_dna.js
    m. select_dna.jsp
    n. select_dna_for_start_plate.js
    o. select_dna_for_start_plate.jsp
    p. select_file.js
    q. select_file.jsp
13. Data access object classes/files Library.java and PooledLibrary.java in src/net/sf/basedb/meludi/dao/ added.
14. Java servlet classes/files DnaServlet.java and LibPrepServlet.java in src/net/sf/basedb/meludi/servlet/ added.

[olle]

(In [3450]) Refs #801. Refs #792. Debug output in library preparation java servlets removed:

1. Java servlet classes/files DnaServlet.java and LibPrepServlet.java in src/net/sf/basedb/meludi/servlet/ updated by removing debug output.

[olle]

(In [3452]) Refs #801. Refs #792. Instruction file README in / updated by increasing minimum BASE version from 3.4 to 3.5 (needed for use of ItemList items).

[olle]

Milestone renamed

[olle]

Ticket closed as MeLuDI version 1.3.1 has been created.