Retraction wizard - Step 3: Destruction

[olle]

Retraction wizard steps:

1. [Ticket #526] Marking extracts for retraction, produce lists and protocols to simplify finding these in the lab or external storage.
2. [Tickets #723, #769] Follow-up of step 1, where a web form is presented for a selected retraction object, allowing dates to be entered when different sub-steps are finished. When all sub-steps are marked to have been finished, the items are marked to be ready for deletion/destruction.

This ticket is concerned with the final step, where a retraction object ready for deletion/destruction is selected, after which a list of items to be deleted, and data to be be kept, is displayed. After approval, the items listed for deletion will be deleted from the database, copies of the case and biosource items will be created, after which the original case and biosource items will also be deleted (biosource here refers to a "patient" item of type Retract or RetroNo). Copies of the last two items are used, in order not to have references to log data for changes of the item data. The reason for a keeping a copy of the case item, is also to prevent its SCAN-B id value to be reused by new items.

[olle]

Ticket accepted.

[olle]

Traceability note:

• The retraction wizard was introduced in Ticket #526 (Retraction wizard), which covered step 1.
• It was updated in Ticket #723 (Retraction wizard - Step 2: Follow up), which covered step 2.
• Retraction wizard step 2 was updated in Ticket #769 (Retraction wizard - Additions to Step 2: Follow up).
• Step 3 of the retraction wizard was introduced in Ticket #770 (Retraction wizard - Step 3: Destruction) - This ticket.
• The retraction wizard servlet was cleaned up in Ticket #781 (Clean up code in retraction wizard)
• The retraction wizard was updated to support BloodDNA in Ticket #787 (Add support for BloodDNA items), where also support for Blood items without any Case item was added.

[olle]

Background:

The full retraction module is intended to contain 3 parts:
1. Mark extracts for retraction. (Ticket #526)
2. Retraction follow-up (Tickets #723 and #769)
3. Destruction (this ticket)

1. When part 1, "Mark extracts for retraction", is finished, the patient type is changed from "Patient" to "Retract" or "RetroNo". However, for simplicity, it will be referred to as a patient item in this description.
2. Biosource items of type "Retract" or "RetroNo" keep track of what parts in the retraction module have been finished through enumerated string annotation "RetractStatus" with options "ReadyForDestruction" and "RetractionCompleted" (null is default; option "RetractionCompleted" is intended to be set in part 3 "Destruction"). When part 2, "Retraction follow-up", is finished, annotation "RetractStatus" is set to value "ReadyForDestruction".

Functional specification for the retraction destruction part:

3. Step 1: A case name or personal number should be input:
   a. If a case name is input, the patient related to that case should be found. As is described in the next section, the "patient" item will technically be a biosource item of type "Retract" or "RetroNo", instead of "Patient", but for for simplicity, it will be referred to as a patient item in this description.
   b. Part 3, "Retraction destruction", is not intended to be used until parts 1 and 2 are finished, so it is natural to include a check that the patient type is correct for retraction ("Retract" or "RetroNo"). In addition, annotation "RetractStatus" should have value "ReadyForDestruction".

[olle]

Description updated by only requiring that a copy of the case item be created, but not the biosource item.

[olle]

Description updated again by reinstating the requirement that a copy of the biosource item be created, but changing the reason to only being that no references to log data are desired.

[olle]

Traceability note:

• The retraction wizard was updated in Ticket #781 (Clean up code in retraction wizard) by combining earlier RetractionFormServlet and RetractionFollowUpFormServlet into a new RetractionServlet, which affects all retraction wizards.

[olle]

(In [3333]) Refs #770. Java class/file RetractionServlet.java in src/net/sf/basedb/reggie/servlet/ updated by removal of unused debug output statements, and update of servlet name in active debug output.

[olle]

(In [3334]) Refs #770. Java class/file RetractionServlet.java in src/net/sf/basedb/reggie/servlet/ updated by removal of dead or unused code.

[olle]

Functional specification for the retraction destruction part:

1. Step 1: A case name or personal number should be input:
   a. If a case name is input, the patient related to that case should be found. The "patient" item will technically be a biosource item of type "Retract" or "RetroNo", instead of "Patient", but for for simplicity, it will be referred to as a patient or biosource item in this description.
   b. Part 3, "Retraction destruction", is not intended to be used until parts 1 and 2 are finished, so a check is included that the patient type is correct for retraction ("Retract" or "RetroNo"). In addition, annotation "RetractStatus" should have value "ReadyForDestruction".
2. Step 2: The following data should be displayed:
   a. The 8 dates registered in the retraction follow-up step for the biosource.
   b. A list of type, name, and comment for the biosource, case(s), and sub-items. The "type" specification for items other than biosource and case is restricted to the main type, such as "Sample" and "Extract". The "comment" field should contain information on what effect a click on the "Register" button will have on the item, such as "To be replaced by a copy with the same name" (for biosources and cases), "To be deleted", or "Retained, but extract ... to be removed from parent list" (for pooled libraries).
   c. A "Cancel" and a "Register" button. Clicking the "Register" button should trigger the following actions: Creation of copies of the biosource and case items, with the same names, but otherwise no information, except that the biosource copy has annotation "RetractStatus" set to value "RetractionCompleted", and that the biosource copy is set as parent to the case item copy/copies. The original biosource and case items, as well as sub-items except pooled libraries should be deleted. The deleted extract items should be removed from the parent list of any pooled library sub-item (this is necessary for the former items to be deleted from the database).

Design update:

1. JSP file index.jsp in resources/ updated in "Retraction wizard" section with new link "Destruction", coupled to new JSP file retraction_destruct_form.jsp in resources/personal/.
2. JSP file retraction_destruct_form.jsp in resources/personal/ added. It is coupled to javascript file retraction_destruct_form.js.
3. Javascript file retraction_destruct_form.js in resources/personal/ added. It calls servlet RetractionServlet with the following commnds:
   a. Function caseNameOnChange(event) calls command "GetCaseInfo" to get information from which to decide, whether the entered case is valid for further processing.
   b. Function initializeStep2(event) calls command "GetCaseInfoBeforeDestruction" to get a list of items related to the entered case name, that needs to be handled.
   c. Function submit() calls command "RegisterDestruction" to perform the actual deletion of items, and other actions needed to complete the retraction.
4. Java servlet class/file RetractionServlet.java in src/net/sf/basedb/reggie/servlet/ updated:
   a. New public static final constant string constant RETRACTION_COMPLETED with value "RetractionCompleted" added.
   b. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated with new command "GetCaseInfoBeforeDestruction". It calls new private method JSONObject fetchJsonCaseInfoBeforeDestruction(DbControl dc, String caseOrPersonalNumber, JSONObject json) to obtain the JSON object to return.
   c. Protected method void doPost(HttpServletRequest req, HttpServletResponse resp) updated with new command "RegisterDestruction". Sub-items are found by calling new private method void createSubItemList(DbControl dc, BasicItem basicItem, List<BasicItem> list). It calls new private method HashMap<String,List<Extract>> fetchPooledLibraryNameParentExtractListHashMap(DbControl dc, List<BasicItem> basicItemList) to obtain a hash map used to find what parent extract items need be removed from a pooled library parent list, in order to complete removal of the former. Copies of biosource and case items are created with the same names, but otherwise no information, except that the biosource copy has annotation "RetractStatus" set to value "RetractionCompleted", and that the biosource copy is set as parent to the case item copy/copies. The original biosource and case items as well as sub-items except pooled libraries are deleted. Pooled libraries are updated by removal of deleted extract items from the parent list.
   d. New private method JSONObject fetchJsonCaseInfoBeforeDestruction(DbControl dc, String caseOrPersonalNumber, JSONObject json) added. It returns an input JSONObject updated with case info data, where sub-items are found using recursion by calling new private method void createSubItemList(DbControl dc, BasicItem basicItem, List<BasicItem> list). It calls new private method HashMap<String,List<Extract>> fetchPooledLibraryNameParentExtractListHashMap(DbControl dc, List<BasicItem> basicItemList) to obtain a hash map used to find what parent extract items need be removed from a pooled library parent list, in order to complete removal of the former.
   e. New private method void createSubItemList(DbControl dc, BasicItem basicItem, List<BasicItem> list) added. It finds all sub-items to a basicItem using method Set<ItemProxy> getUsingItems() and recursion.
   f. New private method HashMap<String,List<Extract>> fetchPooledLibraryNameParentExtractListHashMap(DbControl dc, List<BasicItem> basicItemList) added. It returns a hashmap with pooled library names as keys and parent extract list as value for an input list of basicitems.

[olle]

(In [3337]) Refs #770. Retraction wizard updated with first version of the destruction part:

1. JSP file index.jsp in resources/ updated in "Retraction wizard" section with new link "Destruction", coupled to new JSP file retraction_destruct_form.jsp in resources/personal/.
2. JSP file retraction_destruct_form.jsp in resources/personal/ added. It is coupled to javascript file retraction_destruct_form.js.
3. Javascript file retraction_destruct_form.js in resources/personal/ added. It calls servlet RetractionServlet with the following commnds:
   a. Function caseNameOnChange(event) calls command "GetCaseInfo" to get information from which to decide, whether the entered case is valid for further processing.
   b. Function initializeStep2(event) calls command "GetCaseInfoBeforeDestruction" to get a list of items related to the entered case name, that needs to be handled.
   c. Function submit() calls command "RegisterDestruction" to perform the actual deletion of items, and other actions needed to complete the retraction.
4. Java servlet class/file RetractionServlet.java in src/net/sf/basedb/reggie/servlet/ updated:
   a. New public static final constant string constant RETRACTION_COMPLETED with value "RetractionCompleted" added.
   b. Protected method void doGet(HttpServletRequest req, HttpServletResponse resp) updated with new command "GetCaseInfoBeforeDestruction". It calls new private method JSONObject fetchJsonCaseInfoBeforeDestruction(DbControl dc, String caseOrPersonalNumber, JSONObject json) to obtain the JSON object to return.
   c. Protected method void doPost(HttpServletRequest req, HttpServletResponse resp) updated with new command "RegisterDestruction". Sub-items are found by calling new private method void createSubItemList(DbControl dc, BasicItem basicItem, List<BasicItem> list). It calls new private method HashMap<String,List<Extract>> fetchPooledLibraryNameParentExtractListHashMap(DbControl dc, List<BasicItem> basicItemList) to obtain a hash map used to find what parent extract items need be removed from a pooled library parent list, in order to complete removal of the former. Copies of biosource and case items are created with the same names, but otherwise no information, except that the biosource copy has annotation "RetractStatus" set to value "RetractionCompleted", and that the biosource copy is set as parent to the case item copy/copies. The original biosource and case items as well as sub-items except pooled libraries are deleted. Pooled libraries are updated by removal of deleted extract items from the parent list.
   d. New private method JSONObject fetchJsonCaseInfoBeforeDestruction(DbControl dc, String caseOrPersonalNumber, JSONObject json) added. It returns an input JSONObject updated with case info data, where sub-items are found using recursion by calling new private method void createSubItemList(DbControl dc, BasicItem basicItem, List<BasicItem> list). It calls new private method HashMap<String,List<Extract>> fetchPooledLibraryNameParentExtractListHashMap(DbControl dc, List<BasicItem> basicItemList) to obtain a hash map used to find what parent extract items need be removed from a pooled library parent list, in order to complete removal of the former.
   e. New private method void createSubItemList(DbControl dc, BasicItem basicItem, List<BasicItem> list) added. It finds all sub-items to a basicItem using method Set<ItemProxy> getUsingItems() and recursion.
   f. New private method HashMap<String,List<Extract>> fetchPooledLibraryNameParentExtractListHashMap(DbControl dc, List<BasicItem> basicItemList) added. It returns a hashmap with pooled library names as keys and parent extract list as value for an input list of basicitems.

[olle]

(In [3338]) Refs #770. Retraction wizard updated in destruction part by commenting out statements for debug output, except for exceptions and other conditions, that might prevent the servlet to perform the destruction task:

1. Java servlet class/file RetractionServlet.java in src/net/sf/basedb/reggie/servlet/ updated in destruction part by commenting out statements for debug output, except for exceptions and other conditions, that might prevent the servlet to perform the destruction task.

[Nicklas Nordborg]

(In [3374]) References #770 and #791. The retraction wizard (destruction step) will now check all items for outgoing AnyToAny links. Mainly to find files that are linked to various items (eg. histology images).

Files with have set the "write protected" flag are a bit tricky since they are not possible to delete and re-setting the flag doesn't update the permissions unless DbControl.reattachItem() is called.

[Nicklas Nordborg]

(In [3375]) References #770: Retraction wizard - Step 3: Destruction

Disabling this wizard since it has not been fully tested.

[Nicklas Nordborg]

Ticket retargeted after milestone closed

[Nicklas Nordborg]

(In [3382]) References #770: Retraction wizard - Step 3: Destruction

Enabling this wizard again so we can make a test installation.

[Nicklas Nordborg]

(In [3386]) References #770: Retraction wizard - Step 3: Destruction

The recursive loading of child items must not continue downstream from PooledLibrary items since this will lead to flow cells and all derived bioassays for all libraries in the pool.

[Nicklas Nordborg]

(In [3387]) References #770: Retraction wizard - Step 3: Destruction

Load files that are linked via FileSet:s.

Display the full path to files instead of just the name.

[Nicklas Nordborg]

(In [3388]) References #770: Retraction wizard - Step 3: Destruction

Using a Set instead of List to collect items, since it will automatically avoid duplicates to be inserted.

[Nicklas Nordborg]

(In [3389]) References #770: Retraction wizard - Step 3: Destruction

Adding subtype information to output.

[Nicklas Nordborg]

(In [3390]) References #770: Retraction wizard - Step 3: Destruction

Changed detection of pooled libraries to use the subtype of the items instead of the names.

[Nicklas Nordborg]

(In [3391]) References #770: Retraction wizard - Step 3: Destruction

Finding (secondary analysis) directories that should also be deleted.

[Nicklas Nordborg]

(In [3392]) References #770: Retraction wizard - Step 3: Destruction

Remove unused imports.

[Nicklas Nordborg]

(In [3393]) References #770: Retraction wizard - Step 3: Destruction

Fixing issue that cause the full path to file items to be lost.

[Nicklas Nordborg]

(In [3425]) References #770: Retraction wizard - Step 3: Destruction

Cleaned up a lot of unused code and simplified the loading and handling of child items. Since all child items are found by the collectAllChildItems() method there is really no reason to load cases and blood items separately.

The code for actually deleting items has temporarily been disabled in order to allow for repeated testing.

[Nicklas Nordborg]

(In [3427]) References #770: Retraction wizard - Step 3: Destruction

Checking files on external server (ProjectArchive) and trying to get the top-most directory for a sample.

Sending 'rm -rf' command over SSH to delete the directory found in the check mentioned above.

Checking for DELETE permission on all items and disabling the "Register" button unless all items can be deleted.

[Nicklas Nordborg]

(In [3472]) References #770: Retraction wizard - Step 3: Destruction

Disabling this wizard since it has not been fully tested.

[Nicklas Nordborg]

Ticket retargeted after milestone closed

[Nicklas Nordborg]

Milestone renamed

[Nicklas Nordborg]

(In [3769]) References #770: Retraction wizard - Step 3: Destruction

Setting "Consent" to "No" on the Case copies that are created to act as placeholder for the retracted data.