Registration of specimen handled by external lab

[Nicklas Nordborg]

Note** This has only been fixed for importing from RNA. Further development is needed to support importing from Specimen or Case.

The current registration procedure assumes that we get all specimen tubes to our lab. If we don't get any specimen this is handled by creating a NoSpecimen item.

This is going to change and some specimen will be handled and sequenced by an external lab. We still need to register information about those specimen since we will get the same referral form as before. Processing information and results from the external lab will be imported later by a completely new wizard.

To begin with the "Pathology referral form registration" wizard need modifications so that it is possible to register a specimen as a FutureSpecimen (this is a suggestion). This is an indication that results will come later, and the FutureSpecimen will then transform to a regular !Specimen item (or possible more than one if multiple tubes was used for the same referral). It is also possible that the FutureSpecimen should be transformed to a NoSpecimen in case we never get any results.

This change will most likely affect several of other wizards, such as the consent and blood registration wizard, to make sure that they find the correct information based on a given SCAN-B id.

[Nicklas Nordborg]

In [6151]

References #1295: Registration of specimen handled by external lab

Added Subtype.FUTURE_SPECIMEN. The "Pathology referral form registration" wizard has been updated to allow the user to choose if a NoSpecimen or a FutureSpecimen item should be created if no specimen tubes exits.

Functionality in the Case class that is locating cases by name have been updated to also consider FutureSpecimen items. This means that several other wizards such as the Consent and Blood registration should work as expected without any more changes.

[Nicklas Nordborg]

In 6152:

References #1295: Registration of specimen handled by external lab

Added Subtype.FUTURE_SPECIMEN. The "Pathology referral form registration" wizard has been updated to allow the user to choose if a NoSpecimen or a FutureSpecimen item should be created if no specimen tubes exits.

Functionality in the Case class that is locating cases by name have been updated to also consider FutureSpecimen items. This means that several other wizards such as the Consent and Blood registration should work as expected without any more changes.

[Nicklas Nordborg]

In 6153:

References #1295: Registration of specimen handled by external lab

Updated some export and batch import wizards.

• INCA export, and Monthly operation list export and missing data export also export cases with FutureSpecimen items
• Map external data to releases include PAD and external id from FutureSpecimen
• Import frozen tissue date will find cases based on FutureSpecimen names
• Follow-up blood registration will find cases based on FutureSpecimen names
• Linked specimen and Reference date wizards should handle FutureSpecimen items

[Nicklas Nordborg]

In 6154:

References #1295: Registration of specimen handled by external lab

The case summary has been updated to display information about FutureSpecimen items.

[Nicklas Nordborg]

In 6155:

References #1295: Registration of specimen handled by external lab

FutureSpecimen items are now displayed also in the first retraction step. Otherwise they are handled as NoSpecimen items.

[Nicklas Nordborg]

In [6159]

References #1295: Registration of specimen handled by external lab

Display information about FutureSpecimen on the "Edit" page.

[Nicklas Nordborg]

In 6177:

References #1295: Registration of specimen handled by external lab

Started to work on an import wizard for external specimen. There is not really any functionality yet. The wizard will display existing FutureSpecimen items and try to match them against json files in the ImportArchive. The match is made by name now, but will probably need to parse information from the json file in the future. Json files that doesn't match a FutureSpecimen item will try matching against a specimen item. This should handle the use case where we send an existing specimen tube to the external lab which is more similar to a "Re-extract from specimen" action than a completely new specimen. The wizard should also support re-process from RNA but this will have to wait until we have some real data files.

[Nicklas Nordborg]

In 6178:

References #1295: Registration of specimen handled by external lab

Started to work on an import wizard for external specimen. There is not really any functionality yet. The wizard will display existing FutureSpecimen items and try to match them against json files in the ImportArchive. The match is made by name now, but will probably need to parse information from the json file in the future. Json files that doesn't match a FutureSpecimen item will try matching against a specimen item. This should handle the use case where we send an existing specimen tube to the external lab which is more similar to a "Re-extract from specimen" action than a completely new specimen. The wizard should also support re-process from RNA but this will have to wait until we have some real data files.

[Nicklas Nordborg]

In 6193 and 6194:

References #1295: Registration of specimen handled by external lab

Started to work on the actual importer implementation. It is currently a mockup that only create empty items in the chain from FutureSpecimen down to MergedSequences. Other wizards are likely to fail if used with the items.

Some real changes in the other API have been implemented:

• The ReggieItem.getNextChildItemName() can now generate child item names also for parent items that have not yet been saved in the database. Several related changes in other dao classes.

• Added static get(item) method to several dao classes to make them usable for items that has not been saved in the database.

[Nicklas Nordborg]

In 6195:

References #1295: Registration of specimen handled by external lab

Added Library plate creation.

[Nicklas Nordborg]

In 6197:

References #1295: Registration of specimen handled by external lab

Added ExternalRef annotation intended to be used for storing a reference id generated by the external lab on items that we need to use multiple times for different samples. For example: pool, library plate, flow cell, etc.

[Nicklas Nordborg]

In 6198:

References #1295: Registration of specimen handled by external lab

Re-factoring to prepare for better error handling.

[Nicklas Nordborg]

In 6199:

References #1295: Registration of specimen handled by external lab

Another big re-factoring to separate validation of the JSON file from actual import and errors that can happen there. So far, only code for Library and Pool have been implemented.

[Nicklas Nordborg]

In 6200:

References #1295: Registration of specimen handled by external lab

Another big re-factoring to separate validation of the JSON file from actual import and errors that can happen there. So far, only code for Library and Pool have been implemented.

[Nicklas Nordborg]

In 6201 6202:

References #1295: Registration of specimen handled by external lab

Added code for creating FlowCell and SequencingRun items.

[Nicklas Nordborg]

In 6203:

References #1295: Registration of specimen handled by external lab

Added code for creating DemuxedSequences and MergedSequences items.

[Nicklas Nordborg]

In 6204:

References #1295: Registration of specimen handled by external lab

Added code for creating DemuxedSequences and MergedSequences items.

[Nicklas Nordborg]

In 6205:

References #1295: Registration of specimen handled by external lab

Moved validation of already existing items that can be referenced (eg. Barcode, Pool, Libplate, Flowcell, etc.) to the initial parsing of the JSON file so that things that will cause the import to fail are displayed immediately.

[Nicklas Nordborg]

In 6206:

References #1295: Registration of specimen handled by external lab

Implemented some checks that are displayed as warnings when importing. For example, if a date is in the future or older than 1 year.

[Nicklas Nordborg]

In 6208:

References #1295: Registration of specimen handled by external lab

Added a download link so that it is possible to inspect the JSON file.

[Nicklas Nordborg]

In 6209:

References #1295: Registration of specimen handled by external lab

Filling in some missing values in the import.

Renamed the ImportArchive file server to ImportGateway and created a new ImportArcive. The idea is that the ImportGateway is a temporary gateway were date is waiting to be imported. Once imported the data (JSON and FASTQ files) is copied to the ImportArchive for permanent storage. Secondary analysis is then using the files in the ImportArchive. The ImportFastqJobCreator has been update to handle the moving of FASTQ files. JSON files are not yet handled but I think it has to be done by the importer. It will currently generate a lot of error message if the file is left at the gateway.

[Nicklas Nordborg]

In 6210:

References #1295: Registration of specimen handled by external lab

After a successful import the JSON file is now moved from the ImportGateway to the ImportArchive and the resulting merged bioassay is added to the FASTQ import pipeline. Auto-confirm support remains to be implemented.

[Nicklas Nordborg]

In 6211:

References #1295: Registration of specimen handled by external lab

Implemented support for auto-starting FASTQ import after a successful JSON import.

[Nicklas Nordborg]

In 6212:

References #1295: Registration of specimen handled by external lab

Added possibility to select an "Operator" when importing that override values from the JSON files.

[Nicklas Nordborg]

In 6213:

References #1295: Registration of specimen handled by external lab

Added MergeType and DemuxType annotations to be used on Software/Protocol items to make it possible to separate the external pipeline from the internal.

[Nicklas Nordborg]

In 6214:

References #1295: Registration of specimen handled by external lab

Moved the DataFilesFolder annotation from SequencingRun to DemuxedSequences since that makes more sense when dealing with the external sequencing where we get demuxed FASTQ files. This also makes it possible to implement checks in the "Start demux" and "Import FASTQ" wizards that data in the expected format is available. Eg. a SequencingRun with a DataFilesFolder should contain raw sequencing data and a DemuxedSequences should contain FASTQ files.

[Nicklas Nordborg]

In 6215:

References #1295: Registration of specimen handled by external lab

Re-designed the importer to stop at DemuxedSequnces instead of MergedSequences.

There will be one DemuxedSequences for each library instead of one for each flow cell. The DemuxedSequences item will be a named after the library an use 'x' suffix. The suffix is NOT propagated to child MergedSequences which uses 'g' as before. For example:

Lib: 1234.r.lib
Demux: 1234.r.lib.x
Merge: 1234.r.lib.g

The DemuxedSequences will also have information about the original FASTQ files that we recieved from the external lab.

The MergedSequences item is now created by the FASTQ import script. This change makes it easier to redo failed FASTQ import since we can just delete the MergedSequences and try again.

[Nicklas Nordborg]

In 6216:

References #1295: Registration of specimen handled by external lab

The SCAN-B report failed if the library doesn't have a protocol.

[Nicklas Nordborg]

In 6217:

References #1295: Registration of specimen handled by external lab

Started to implement the specimen-specific details.

[Nicklas Nordborg]

In 6221:

References #1295: Registration of specimen handled by external lab

Preparations for importing protocol info assuming that we can match the provided value to the ExternalRef annotation of a protocol.

Also some preparations for MD5 if we will get that for the FASTQ files.

[Nicklas Nordborg]

In 6222:

References #1295: Registration of specimen handled by external lab

Preparations for importing protocol info assuming that we can match the provided value to the ExternalRef annotation of a protocol.

Also some preparations for MD5 if we will get that for the FASTQ files.

[Nicklas Nordborg]

In 6239:

References #1295: Registration of specimen handled by external lab

The "Other path note" field should be hidden if there are already registred specimen.

[Nicklas Nordborg]

In 6240:

References #1295: Registration of specimen handled by external lab

Changed ref in debug messages to other more descriptive names, for example, ExternalRef or FlowCellID.

[Nicklas Nordborg]

In 6338:

References #1295: Registration of specimen handled by external lab

Fixed a bug in the protocol validation.

[Nicklas Nordborg]

In 6341:

References #1295: Registration of specimen handled by external lab

Implemented support for finding a root item to link to when importing. An ID should be available in "SCANB_ID" attribute on the main object. This ID should resolve to either a (pre-normalized) RNA, a Specimen or a Case/FutureSpecimen.

Only the first two cases are considered so far.

[Nicklas Nordborg]

In 6342:

References #1295: Registration of specimen handled by external lab

Implemented support for finding a root item to link to when importing. An ID should be available in "SCANB_ID" attribute on the main object. This ID should resolve to either a (pre-normalized) RNA, a Specimen or a Case/FutureSpecimen.

Only the first two cases are considered so far.

[Nicklas Nordborg]

In 6343:

References #1295: Registration of specimen handled by external lab

Changed date format in Sequencing run StartDate and EndDate.

[Nicklas Nordborg]

In 6344:

References #1295: Registration of specimen handled by external lab

Get RNA and DNA concentration from the JSON file.

[Nicklas Nordborg]

In 6346:

References #1295: Registration of specimen handled by external lab

Added support for implementing checks that verify that information that is expected to be unique is not duplicated in more than one file. Two checks have been implemented so far:

• That a well on a library plate is only used once
• That a barcode is only used once in a pool

[Nicklas Nordborg]

In 6347:

References #1295: Registration of specimen handled by external lab

Added support for implementing checks that verify that information that is expected to be unique is not duplicated in more than one file. Two checks have been implemented so far:

• That a well on a library plate is only used once
• That a barcode is only used once in a pool

[Nicklas Nordborg]

In 6348:

References #1295: Registration of specimen handled by external lab

Disabled importing to items except pre-normalized RNA. We want to make a release that at least can import some of the current data and will have to delay the more complicated cases until later.

[Nicklas Nordborg]

In 6349:

References #1295: Registration of specimen handled by external lab

Disabled the option to create a FutureSpecimen item in the Pathology referral form wizard since we do not yet support importing data to them.

[Nicklas Nordborg]

In 6350:

References #1295: Registration of specimen handled by external lab

A mistting title tag caused the first step to disappear instead of being minimized.

[Nicklas Nordborg]

In 6351:

References #1295: Registration of specimen handled by external lab

Removing special ID replacements that should be fixed on the sending end (or by manual editing of already existing files).

[Nicklas Nordborg]

In 6352:

References #1295: Registration of specimen handled by external lab

Only output debug messages if the debug flag is set.

[Nicklas Nordborg]

In 6353:

References #1295: Registration of specimen handled by external lab

Sort items by date of the JSON file.