Updates to registration of specimen handled by external lab

[Nicklas Nordborg]

See #1349, #1347 and #1295 for background information about earlier updates. This time the following things are needed:

• Create Tag items from JSON. Barcoding scheme has changed to IDT10 UDI. There are close to 4000 different combinations. Instead of creating them all beforehand, it should be possible to create new Tag items when needed. The barcode in the JSON file is typically of the format: 03B IDT10_UDI_Adapter2018 (ATCTGGCCAT-CCGCTGTCTA). The first part (03B) can be ignored, the second part IDT10_UDI_Adapter2018 should be the name of the barcode, and the two sequences are the BacodeSequence/BarcodeSequence2 annotations.

• I see some warnings about Old date in JSON for partition date, extraction date, etc. and a reference to another date that is not found in the JSON. For example: Old date in JSON: Lysate.Partition date=2021-11-11 (expected after 2021-11-24). 2021-11-24 seems to be the registration date of the Patient item which doesn't make sense. This should be investigated.

• The queue with items to import is very long now. This could potentially be problematic since the GUI is not optimized for this. It may take a very long time to load and check all JSON files and if that doesn't fail it will be difficult to select/de-select items for importing.

[Nicklas Nordborg]

In 6887:

References #1420: Updates to registration of specimen handled by external lab

Barcodes should now be created as needed.

[Nicklas Nordborg]

In 6888:

References #1420: Updates to registration of specimen handled by external lab

Added information about item type to debug messages.

[Nicklas Nordborg]

In 6893:

References #1420: Updates to registration of specimen handled by external lab

This should take care of most issues with the "Old date" warnings. It turned out that there was a default of max 1 year old dates. This default is still kept, but if there is a pre-registered NoSpecimen item already the sampling date from that item is used instead. There is also an option to use the reference date from the Case item, but it is not likely to exist either if the NoSpecimen item doesn't have a date.

[Nicklas Nordborg]

In 6894:

References #1420: Updates to registration of specimen handled by external lab

Information about existing FASTQ files are now loaded by bulk with a find command instead of doing it one by one.

[Nicklas Nordborg]

In 6895:

References #1420: Updates to registration of specimen handled by external lab

Started to implement support for only checking a smaller number of JSON files to begin with. The idea is to be able to manually validate more files, but that is not yet implemented.

[Nicklas Nordborg]

In 6896:

References #1420: Updates to registration of specimen handled by external lab

Implemented functionality manually validating files that was not automatically validated when loading the original page.

[Nicklas Nordborg]

In 6897:

References #1420: Updates to registration of specimen handled by external lab

Try to get flowcell ID of continue to validate JSON files with the same flowcell ID even if the max number has been reached.

The list is also re-orginised so that JSON files that has been validated are sorted first and JSON files with the same flowcell ID are sorted together.

[Nicklas Nordborg]

In 6898:

References #1420: Updates to registration of specimen handled by external lab

Added a progress reporter that shows the number of JSON files that have been parsed.

[Nicklas Nordborg]

In 6899:

References #1420: Updates to registration of specimen handled by external lab

Implemented a function for validating all JSON files. The default max is now 30 which is probably a reasonable value in the future.

[Nicklas Nordborg]

In 6900:

References #1420: Updates to registration of specimen handled by external lab

Added a check that the same FASTQ files are not referenced in more than one JSON file.

[Nicklas Nordborg]

In 6901:

References #1420: Updates to registration of specimen handled by external lab

Added an option to show less of the warning and error messages to get a somewhat better overview of the list.

[Nicklas Nordborg]

In 6902:

References #1420: Updates to registration of specimen handled by external lab

Strip white-space from the SCANB_ID values. There is an extra \r at the end in some of the files.

[Nicklas Nordborg]

In 6903:

References #1420: Updates to registration of specimen handled by external lab

Disabling code that check for a box name and position since we know that this will not be in any of the files.

[Nicklas Nordborg]

In 6904:

References #1420: Updates to registration of specimen handled by external lab

Skip warning for missing values for the following Specimen fields since it is missing in all JSON files:

• Laterality
• Sampling date
• RNA later date
• Number of tubes

Add a warning message if both the value in JSON and on a linked specimen is null.

Add an option to not warn when copying a value from an existing specimen when there is no value in the JSON file.

The first 3 fields above should (hopefully) be registered from the referral form already so it makes sense to only warn if they are missing in both places.

[Nicklas Nordborg]

In 6905:

References #1420: Updates to registration of specimen handled by external lab

Minor changes in visibility of the number of warnings and the 'Show more'/'Show less' functionality.

[Nicklas Nordborg]

In 6906:

References #1420: Updates to registration of specimen handled by external lab

Date validation was not strict enough. There was a file with a date value 20-30-01-01 which was parsed to 0022-06-01 (eg. year=20, month=30, day=1). There was a warning about this being an old date, but with stricter parsing an error is generated instead.

[Nicklas Nordborg]

In 6907:

References #1420: Updates to registration of specimen handled by external lab

Only display the parsed result of a date value if it is different from the string value in the JSON file.

[Nicklas Nordborg]

In 6908:

References #1420: Updates to registration of specimen handled by external lab

Always try to load file metadata even if there is another error.

[Nicklas Nordborg]

In 6909:

References #1420: Updates to registration of specimen handled by external lab

Fixes a NullPointerException when importing.

[Nicklas Nordborg]

In 6910:

References #1420: Updates to registration of specimen handled by external lab

Added a new annotation ImportWarnings for Specimen items (and also RNA in case we do that in the future). All warnings during the import are saved to that annotation.

[Nicklas Nordborg]

In 6911:

References #1420: Updates to registration of specimen handled by external lab

Sort by sequencing start date before flow cell id and file date.

[Nicklas Nordborg]

In 6912:

References #1420: Updates to registration of specimen handled by external lab

Extend default reference date to 500 days back since we now have a really big backlog of data.

[Nicklas Nordborg]

In 6913:

References #1420: Updates to registration of specimen handled by external lab

Added a progress reporter for the import.

[Nicklas Nordborg]

In 6914:

References #1420: Updates to registration of specimen handled by external lab

Do not display all warnings after importing since it may create a very long list. Added possibility to include the warnings if the debug flag is used in the script.

[Nicklas Nordborg]

In 6915:

References #1420: Updates to registration of specimen handled by external lab

Report the number of JSON files that can be importer but with warnings.

[Nicklas Nordborg]

In 6916:

References #1420: Updates to registration of specimen handled by external lab

Changed the error that was reporter if Specimen.Remaining quantity < 0 to a warning.

Also add a new warning if Lysate.Used quantity > Specimen.Original quantity.

[Nicklas Nordborg]

In 6917:

References #1420: Updates to registration of specimen handled by external lab

Implemented a method for finding out which existing FASTQ files that has not been named in the JSON files.

[Nicklas Nordborg]

In 6918:

References #1420: Updates to registration of specimen handled by external lab

Added annotation ExternalStorage and use it for storing the *.Storage location value from JSON files for Specimen, Lysate, RNA, DNA and FlowThrough.

[Nicklas Nordborg]

In 6919:

References #1420: Updates to registration of specimen handled by external lab

Fixes an issue with comparing dates with and without time part.

[Nicklas Nordborg]

In 6920:

References #1420: Updates to registration of specimen handled by external lab

Added an option to not warn for missing information when copying from an existing sample. Enabled this option for the 'Number of tubes' value.

[Nicklas Nordborg]

In 6921:

References #1420: Updates to registration of specimen handled by external lab

Do not issue a warning if an existing NoSpecimen item already exists since that is the most common case at the moment.

Added some more information to the top of the table about existing and new items.

[Nicklas Nordborg]

In 6922:

References #1420: Updates to registration of specimen handled by external lab

Remove debug output from date warnings.

[Nicklas Nordborg]

In 6923:

References #1420: Updates to registration of specimen handled by external lab

Added a check if a given PAD is found on an existing and unrelated sample. This is most likely something that has been incorrectly registered. (The case I found was an incorrect digit in the SCANB_ID).

[Nicklas Nordborg]

In 6924:

References #1420: Updates to registration of specimen handled by external lab

Added the number of NoSpecimen items that has not been matched to a JSON file to the summary.

Displaty "Other path note" in the list.

[Nicklas Nordborg]

In 6925:

References #1420: Updates to registration of specimen handled by external lab

Implemented a special case of validation for PAD values where a suffix is allowed on the value from the JSON file if the first part matches with the PAD on the NoSpecimen.

[Nicklas Nordborg]

In 6926:

References #1420: Updates to registration of specimen handled by external lab

Add a warning when the PAD is changed so that it is recorded somwhere.

Also changed the ImportWarnings annotation to only be visible for the PatientCurator role since it may contain sensitive information.

[Nicklas Nordborg]

In 6927:

References #1420: Updates to registration of specimen handled by external lab

Display the "Other path note" annotation from the "copy data from"-item.

[Nicklas Nordborg]

In 6928:

References #1420: Updates to registration of specimen handled by external lab

Make sure that existing data is not removed if there is no data in the JSON file.

[Nicklas Nordborg]

In 6929:

References #1420: Updates to registration of specimen handled by external lab

Also include information about all FASTQ files that could not be matched to any JSON file.