Release importer

[Nicklas Nordborg]

A release importer plug-in or wizard needs to be implemented. It should be able to import a release generated from the Reggie Release exporter plug-in. It should look for the files on a FileServer item.

[Nicklas Nordborg]

(In [4366]) References #940: Release importer

Started to work on a release importer plug-in. It connects to a selected FileServer item (we use the generic SSH functionality in the Open Grid extension) and looks for files that the exporter (in Reggie) is supposed to create. Currently it will only count the number of *.json files and check for the exportcomplete file. Once found the plug-in exits.

[Nicklas Nordborg]

(In [4368]) References #940: Release importer

The release import plug-in now load and parse all JSON files and create a few items (samples and extracts). So far, only the name and project key properties are set.

This code should be considered proof-of-concept only, and need a lot of re-factoring before it is usable in practice.

[Nicklas Nordborg]

(In [4370]) References #940: Release importer

The release importer now set the subtype and parent items (for samples and extracts).

[Nicklas Nordborg]

(In [4372]) References #940: Release importer

Started to refactor the release importer so that each item type is handled by it's own class. It feels a bit outdated already, but it should work with the current exporter.

[Nicklas Nordborg]

(In [4381]) References #940: Release importer

Added support for importing annotation types. Annotation types are created and shared to the active project in the same manner as cohort items. This means that annotation types are only visible in projects(=releases) that they are part of. The project-specific annotation flag is not yet handled.

Parts of the code related to the remote file server has been temporarily disabled or modified to make the plug-in work with files that are located on local file system instead due to not having (easy) access to a real file server at the moment.

[Nicklas Nordborg]

(In [4383]) References #940: Release importer

Now reads the files.json (see [4382]) to get the list of cohort data files.

[Nicklas Nordborg]

(In [4384]) References #940: Release importer

Added support for importing annotation values.

Switched to BASE 3.11 since we need the new API for handling project annotations.

[Nicklas Nordborg]

(In [4386]) References #940: Release importer

Started to clean up the code. Error handling has been improved. The code should now be ready for the more advanced stuff that need to be imported.

[Nicklas Nordborg]

(In [4387]) References #940: Release importer

The plug-in now checks that a project is active before it is used and validates that the "Export ID" parameter only contains valid characters.

[Nicklas Nordborg]

(In [4388]) References #940: Release importer

Moved some of the annotation type importer code.

[Nicklas Nordborg]

(In [4390]) References #940: Release importer

Support for setting the project-specific annotations flag when importing annotations.

[Nicklas Nordborg]

(In [4391]) References #940: Release importer

Revert back to using real file server and SSH again.

[Nicklas Nordborg]

(In [4393]) References #940: Release importer

More cleanup.

[Nicklas Nordborg]

(In [4395]) References #940: Release importer

Importing the registration date.

[Nicklas Nordborg]

(In [4397]) References #940: Release importer

Importing the creation date.

[Nicklas Nordborg]

(In [4399]) References #940: Release importer

Added MergedSequences subtype.

[Nicklas Nordborg]

(In [4400]) References #940: Release importer

Added MergedSequences subtype.

[Nicklas Nordborg]

(In [4403]) References #940: Release importer

Importing the DataFilesFolder annotation. We need special handling for this annotation. When creating a new item the default value is always set to the original path. Project-specific values are always created using the release version as a top-level prefix.

For example:

Default value: /S000006/l.r.lib.g/k.a/t
Release 0.9:   /0.9/S000006/l.r.lib.g/k.a/t

[Nicklas Nordborg]

(In [4405]) References #940: Release importer

Importing and creating files in the BASE file system. The files are not yet linked to the owning item.

[Nicklas Nordborg]

(In [4411]) References #940: Release importer

Importing from the new typedefs.json file. Support for importing data file types. Linking files to items.

[Nicklas Nordborg]

(In [4413]) References #940: Release importer

Importing files that are linked with any-to-any links.

[Nicklas Nordborg]

(In [4429]) References #940: Release importer

The release version number is now used for importing instead of the UUID.

[Nicklas Nordborg]

(In [4433]) References #940: Release importer

Removed debug code.

Default value for file server is set to the ReleaseArchive server.

Default value for release version is extracted from the project name (if it contains a version-like pattern).

[Nicklas Nordborg]

(In [4436]) References #940: Release importer

Raw bioassays are created with the raw data type, platform or platform variant specified in the exported file.

[Nicklas Nordborg]

(In [4440]) References #940: Release importer

Added support for resuming the import after a system restart.

[Nicklas Nordborg]

(In [4441]) References #940: Release importer

Added some counters used for a final summary message.

[Nicklas Nordborg]

(In [4445]) References #940: Release importer

Added support for pausing the import.

[Nicklas Nordborg]

(In [4446]) References #940: Release importer

The importer now has support for adding annotations to existing items even if the annotation type is not marked as project-specific. The use case is that the list of annotation types has changed between two releases. Annotations that are new in the later relase are allowed to be set on already existing items. This will not affect the previous release since those annotations are not visible in that release (except if the logged in user is an admin).

In this sense we consider an annotation type to be "new" if it (after importing it) is only shared to the current project.

[Nicklas Nordborg]

(In [4447]) References #940: Release importer

Resuming after a pause should now be implemented in a more stable manner. There was a risk that the same file was processed twice since there was a rather large gap between the transaction commit and the change of the breakpoint to the next file.

Re-configuration after a pause now checks if the relase version parameter has changed and forces a restart from the beginning in that case since the stored state information is no longer valid.

[Nicklas Nordborg]

(In [4463]) References #940: Release importer

Added support for importing annotation types with units.

[Nicklas Nordborg]

(In [4467]) References #940: Release importer

Importing lookup values from the batch-index-lookup.json. Annotation types that matches a lookup in this file have their values converted to the mapped value. See also [4466].

[Nicklas Nordborg]

(In [4486]) References #940: Release importer

Added support for producing a log file with information that may or may not be so useful.

[Nicklas Nordborg]

(In [4487]) References #940: Release importer

The release import plug-in now ends with a "Done" message.

[Nicklas Nordborg]

(In [4508]) References #940: Release importer

Updated opengrid.jar since it it has now been released (Open Grid 1.1).