#1420 closed enhancement (fixed)

Updates to registration of specimen handled by external lab

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: critical Milestone: Reggie v4.41
Component: net.sf.basedb.reggie Keywords:
Cc:

Description

See #1349, #1347 and #1295 for background information about earlier updates. This time the following things are needed:

  • Create Tag items from JSON. Barcoding scheme has changed to IDT10 UDI. There are close to 4000 different combinations. Instead of creating them all beforehand, it should be possible to create new Tag items when needed. The barcode in the JSON file is typically of the format: 03B IDT10_UDI_Adapter2018 (ATCTGGCCAT-CCGCTGTCTA). The first part (03B) can be ignored, the second part IDT10_UDI_Adapter2018 should be the name of the barcode, and the two sequences are the BacodeSequence/BarcodeSequence2 annotations.
  • I see some warnings about Old date in JSON for partition date, extraction date, etc. and a reference to another date that is not found in the JSON. For example: Old date in JSON: Lysate.Partition date=2021-11-11 (expected after 2021-11-24). 2021-11-24 seems to be the registration date of the Patient item which doesn't make sense. This should be investigated.
  • The queue with items to import is very long now. This could potentially be problematic since the GUI is not optimized for this. It may take a very long time to load and check all JSON files and if that doesn't fail it will be difficult to select/de-select items for importing.

Change History (40)

comment:1 by Nicklas Nordborg, 16 months ago

In 6887:

References #1420: Updates to registration of specimen handled by external lab

Barcodes should now be created as needed.

comment:2 by Nicklas Nordborg, 16 months ago

In 6888:

References #1420: Updates to registration of specimen handled by external lab

Added information about item type to debug messages.

comment:3 by Nicklas Nordborg, 15 months ago

In 6893:

References #1420: Updates to registration of specimen handled by external lab

This should take care of most issues with the "Old date" warnings. It turned out that there was a default of max 1 year old dates. This default is still kept, but if there is a pre-registered NoSpecimen item already the sampling date from that item is used instead. There is also an option to use the reference date from the Case item, but it is not likely to exist either if the NoSpecimen item doesn't have a date.

comment:4 by Nicklas Nordborg, 15 months ago

In 6894:

References #1420: Updates to registration of specimen handled by external lab

Information about existing FASTQ files are now loaded by bulk with a find command instead of doing it one by one.

comment:5 by Nicklas Nordborg, 15 months ago

In 6895:

References #1420: Updates to registration of specimen handled by external lab

Started to implement support for only checking a smaller number of JSON files to begin with. The idea is to be able to manually validate more files, but that is not yet implemented.

comment:6 by Nicklas Nordborg, 15 months ago

In 6896:

References #1420: Updates to registration of specimen handled by external lab

Implemented functionality manually validating files that was not automatically validated when loading the original page.

comment:7 by Nicklas Nordborg, 15 months ago

In 6897:

References #1420: Updates to registration of specimen handled by external lab

Try to get flowcell ID of continue to validate JSON files with the same flowcell ID even if the max number has been reached.

The list is also re-orginised so that JSON files that has been validated are sorted first and JSON files with the same flowcell ID are sorted together.

comment:8 by Nicklas Nordborg, 15 months ago

In 6898:

References #1420: Updates to registration of specimen handled by external lab

Added a progress reporter that shows the number of JSON files that have been parsed.

comment:9 by Nicklas Nordborg, 15 months ago

In 6899:

References #1420: Updates to registration of specimen handled by external lab

Implemented a function for validating all JSON files. The default max is now 30 which is probably a reasonable value in the future.

comment:10 by Nicklas Nordborg, 15 months ago

In 6900:

References #1420: Updates to registration of specimen handled by external lab

Added a check that the same FASTQ files are not referenced in more than one JSON file.

comment:11 by Nicklas Nordborg, 15 months ago

In 6901:

References #1420: Updates to registration of specimen handled by external lab

Added an option to show less of the warning and error messages to get a somewhat better overview of the list.

comment:12 by Nicklas Nordborg, 15 months ago

In 6902:

References #1420: Updates to registration of specimen handled by external lab

Strip white-space from the SCANB_ID values. There is an extra \r at the end in some of the files.

comment:13 by Nicklas Nordborg, 15 months ago

In 6903:

References #1420: Updates to registration of specimen handled by external lab

Disabling code that check for a box name and position since we know that this will not be in any of the files.

comment:14 by Nicklas Nordborg, 15 months ago

In 6904:

References #1420: Updates to registration of specimen handled by external lab

Skip warning for missing values for the following Specimen fields since it is missing in all JSON files:

  • Laterality
  • Sampling date
  • RNA later date
  • Number of tubes

Add a warning message if both the value in JSON and on a linked specimen is null.

Add an option to not warn when copying a value from an existing specimen when there is no value in the JSON file.

The first 3 fields above should (hopefully) be registered from the referral form already so it makes sense to only warn if they are missing in both places.

comment:15 by Nicklas Nordborg, 15 months ago

In 6905:

References #1420: Updates to registration of specimen handled by external lab

Minor changes in visibility of the number of warnings and the 'Show more'/'Show less' functionality.

comment:16 by Nicklas Nordborg, 15 months ago

In 6906:

References #1420: Updates to registration of specimen handled by external lab

Date validation was not strict enough. There was a file with a date value 20-30-01-01 which was parsed to 0022-06-01 (eg. year=20, month=30, day=1). There was a warning about this being an old date, but with stricter parsing an error is generated instead.

comment:17 by Nicklas Nordborg, 15 months ago

In 6907:

References #1420: Updates to registration of specimen handled by external lab

Only display the parsed result of a date value if it is different from the string value in the JSON file.

comment:18 by Nicklas Nordborg, 15 months ago

In 6908:

References #1420: Updates to registration of specimen handled by external lab

Always try to load file metadata even if there is another error.

comment:19 by Nicklas Nordborg, 15 months ago

In 6909:

References #1420: Updates to registration of specimen handled by external lab

Fixes a NullPointerException when importing.

comment:20 by Nicklas Nordborg, 15 months ago

In 6910:

References #1420: Updates to registration of specimen handled by external lab

Added a new annotation ImportWarnings for Specimen items (and also RNA in case we do that in the future). All warnings during the import are saved to that annotation.

comment:21 by Nicklas Nordborg, 15 months ago

In 6911:

References #1420: Updates to registration of specimen handled by external lab

Sort by sequencing start date before flow cell id and file date.

comment:22 by Nicklas Nordborg, 15 months ago

In 6912:

References #1420: Updates to registration of specimen handled by external lab

Extend default reference date to 500 days back since we now have a really big backlog of data.

comment:23 by Nicklas Nordborg, 15 months ago

In 6913:

References #1420: Updates to registration of specimen handled by external lab

Added a progress reporter for the import.

comment:24 by Nicklas Nordborg, 15 months ago

In 6914:

References #1420: Updates to registration of specimen handled by external lab

Do not display all warnings after importing since it may create a very long list. Added possibility to include the warnings if the debug flag is used in the script.

comment:25 by Nicklas Nordborg, 15 months ago

In 6915:

References #1420: Updates to registration of specimen handled by external lab

Report the number of JSON files that can be importer but with warnings.

comment:26 by Nicklas Nordborg, 15 months ago

In 6916:

References #1420: Updates to registration of specimen handled by external lab

Changed the error that was reporter if Specimen.Remaining quantity < 0 to a warning.

Also add a new warning if Lysate.Used quantity > Specimen.Original quantity.

comment:27 by Nicklas Nordborg, 15 months ago

In 6917:

References #1420: Updates to registration of specimen handled by external lab

Implemented a method for finding out which existing FASTQ files that has not been named in the JSON files.

comment:28 by Nicklas Nordborg, 15 months ago

In 6918:

References #1420: Updates to registration of specimen handled by external lab

Added annotation ExternalStorage and use it for storing the *.Storage location value from JSON files for Specimen, Lysate, RNA, DNA and FlowThrough.

comment:29 by Nicklas Nordborg, 15 months ago

In 6919:

References #1420: Updates to registration of specimen handled by external lab

Fixes an issue with comparing dates with and without time part.

comment:30 by Nicklas Nordborg, 15 months ago

In 6920:

References #1420: Updates to registration of specimen handled by external lab

Added an option to not warn for missing information when copying from an existing sample. Enabled this option for the 'Number of tubes' value.

comment:31 by Nicklas Nordborg, 15 months ago

In 6921:

References #1420: Updates to registration of specimen handled by external lab

Do not issue a warning if an existing NoSpecimen item already exists since that is the most common case at the moment.

Added some more information to the top of the table about existing and new items.

comment:32 by Nicklas Nordborg, 15 months ago

In 6922:

References #1420: Updates to registration of specimen handled by external lab

Remove debug output from date warnings.

comment:33 by Nicklas Nordborg, 15 months ago

In 6923:

References #1420: Updates to registration of specimen handled by external lab

Added a check if a given PAD is found on an existing and unrelated sample. This is most likely something that has been incorrectly registered. (The case I found was an incorrect digit in the SCANB_ID).

comment:34 by Nicklas Nordborg, 15 months ago

In 6924:

References #1420: Updates to registration of specimen handled by external lab

Added the number of NoSpecimen items that has not been matched to a JSON file to the summary.

Displaty "Other path note" in the list.

comment:35 by Nicklas Nordborg, 15 months ago

In 6925:

References #1420: Updates to registration of specimen handled by external lab

Implemented a special case of validation for PAD values where a suffix is allowed on the value from the JSON file if the first part matches with the PAD on the NoSpecimen.

comment:36 by Nicklas Nordborg, 15 months ago

In 6926:

References #1420: Updates to registration of specimen handled by external lab

Add a warning when the PAD is changed so that it is recorded somwhere.

Also changed the ImportWarnings annotation to only be visible for the PatientCurator role since it may contain sensitive information.

comment:37 by Nicklas Nordborg, 15 months ago

In 6927:

References #1420: Updates to registration of specimen handled by external lab

Display the "Other path note" annotation from the "copy data from"-item.

comment:38 by Nicklas Nordborg, 15 months ago

In 6928:

References #1420: Updates to registration of specimen handled by external lab

Make sure that existing data is not removed if there is no data in the JSON file.

comment:39 by Nicklas Nordborg, 15 months ago

In 6929:

References #1420: Updates to registration of specimen handled by external lab

Also include information about all FASTQ files that could not be matched to any JSON file.

comment:40 by Nicklas Nordborg, 15 months ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.