Opened 7 years ago

Closed 7 years ago

#940 closed task (fixed)

Release importer

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: critical Milestone: Relax v1.0
Component: net.sf.basedb.relax Keywords:
Cc:

Description

A release importer plug-in or wizard needs to be implemented. It should be able to import a release generated from the Reggie Release exporter plug-in. It should look for the files on a FileServer item.

Change History (36)

comment:1 by Nicklas Nordborg, 7 years ago

Owner: set to Nicklas Nordborg
Status: newassigned

comment:2 by Nicklas Nordborg, 7 years ago

(In [4366]) References #940: Release importer

Started to work on a release importer plug-in. It connects to a selected FileServer item (we use the generic SSH functionality in the Open Grid extension) and looks for files that the exporter (in Reggie) is supposed to create. Currently it will only count the number of *.json files and check for the exportcomplete file. Once found the plug-in exits.

comment:3 by Nicklas Nordborg, 7 years ago

(In [4368]) References #940: Release importer

The release import plug-in now load and parse all JSON files and create a few items (samples and extracts). So far, only the name and project key properties are set.

This code should be considered proof-of-concept only, and need a lot of re-factoring before it is usable in practice.

comment:4 by Nicklas Nordborg, 7 years ago

(In [4370]) References #940: Release importer

The release importer now set the subtype and parent items (for samples and extracts).

comment:5 by Nicklas Nordborg, 7 years ago

(In [4372]) References #940: Release importer

Started to refactor the release importer so that each item type is handled by it's own class. It feels a bit outdated already, but it should work with the current exporter.

comment:6 by Nicklas Nordborg, 7 years ago

(In [4381]) References #940: Release importer

Added support for importing annotation types. Annotation types are created and shared to the active project in the same manner as cohort items. This means that annotation types are only visible in projects(=releases) that they are part of. The project-specific annotation flag is not yet handled.

Parts of the code related to the remote file server has been temporarily disabled or modified to make the plug-in work with files that are located on local file system instead due to not having (easy) access to a real file server at the moment.

comment:7 by Nicklas Nordborg, 7 years ago

(In [4383]) References #940: Release importer

Now reads the files.json (see [4382]) to get the list of cohort data files.

comment:8 by Nicklas Nordborg, 7 years ago

(In [4384]) References #940: Release importer

Added support for importing annotation values.

Switched to BASE 3.11 since we need the new API for handling project annotations.

comment:9 by Nicklas Nordborg, 7 years ago

(In [4386]) References #940: Release importer

Started to clean up the code. Error handling has been improved. The code should now be ready for the more advanced stuff that need to be imported.

comment:10 by Nicklas Nordborg, 7 years ago

(In [4387]) References #940: Release importer

The plug-in now checks that a project is active before it is used and validates that the "Export ID" parameter only contains valid characters.

comment:11 by Nicklas Nordborg, 7 years ago

(In [4388]) References #940: Release importer

Moved some of the annotation type importer code.

comment:12 by Nicklas Nordborg, 7 years ago

(In [4390]) References #940: Release importer

Support for setting the project-specific annotations flag when importing annotations.

comment:13 by Nicklas Nordborg, 7 years ago

(In [4391]) References #940: Release importer

Revert back to using real file server and SSH again.

comment:14 by Nicklas Nordborg, 7 years ago

(In [4393]) References #940: Release importer

More cleanup.

comment:15 by Nicklas Nordborg, 7 years ago

(In [4395]) References #940: Release importer

Importing the registration date.

comment:16 by Nicklas Nordborg, 7 years ago

(In [4397]) References #940: Release importer

Importing the creation date.

comment:17 by Nicklas Nordborg, 7 years ago

(In [4399]) References #940: Release importer

Added MergedSequences subtype.

comment:18 by Nicklas Nordborg, 7 years ago

(In [4400]) References #940: Release importer

Added MergedSequences subtype.

comment:19 by Nicklas Nordborg, 7 years ago

(In [4403]) References #940: Release importer

Importing the DataFilesFolder annotation. We need special handling for this annotation. When creating a new item the default value is always set to the original path. Project-specific values are always created using the release version as a top-level prefix.

For example:

Default value: /S000006/l.r.lib.g/k.a/t
Release 0.9:   /0.9/S000006/l.r.lib.g/k.a/t

comment:20 by Nicklas Nordborg, 7 years ago

(In [4405]) References #940: Release importer

Importing and creating files in the BASE file system. The files are not yet linked to the owning item.

comment:21 by Nicklas Nordborg, 7 years ago

(In [4411]) References #940: Release importer

Importing from the new typedefs.json file. Support for importing data file types. Linking files to items.

comment:22 by Nicklas Nordborg, 7 years ago

(In [4413]) References #940: Release importer

Importing files that are linked with any-to-any links.

comment:23 by Nicklas Nordborg, 7 years ago

(In [4429]) References #940: Release importer

The release version number is now used for importing instead of the UUID.

comment:24 by Nicklas Nordborg, 7 years ago

(In [4433]) References #940: Release importer

Removed debug code.

Default value for file server is set to the ReleaseArchive server.

Default value for release version is extracted from the project name (if it contains a version-like pattern).

comment:25 by Nicklas Nordborg, 7 years ago

(In [4436]) References #940: Release importer

Raw bioassays are created with the raw data type, platform or platform variant specified in the exported file.

comment:26 by Nicklas Nordborg, 7 years ago

(In [4440]) References #940: Release importer

Added support for resuming the import after a system restart.

comment:27 by Nicklas Nordborg, 7 years ago

(In [4441]) References #940: Release importer

Added some counters used for a final summary message.

comment:28 by Nicklas Nordborg, 7 years ago

(In [4445]) References #940: Release importer

Added support for pausing the import.

comment:29 by Nicklas Nordborg, 7 years ago

(In [4446]) References #940: Release importer

The importer now has support for adding annotations to existing items even if the annotation type is not marked as project-specific. The use case is that the list of annotation types has changed between two releases. Annotations that are new in the later relase are allowed to be set on already existing items. This will not affect the previous release since those annotations are not visible in that release (except if the logged in user is an admin).

In this sense we consider an annotation type to be "new" if it (after importing it) is only shared to the current project.

comment:30 by Nicklas Nordborg, 7 years ago

(In [4447]) References #940: Release importer

Resuming after a pause should now be implemented in a more stable manner. There was a risk that the same file was processed twice since there was a rather large gap between the transaction commit and the change of the breakpoint to the next file.

Re-configuration after a pause now checks if the relase version parameter has changed and forces a restart from the beginning in that case since the stored state information is no longer valid.

comment:31 by Nicklas Nordborg, 7 years ago

(In [4463]) References #940: Release importer

Added support for importing annotation types with units.

comment:32 by Nicklas Nordborg, 7 years ago

(In [4467]) References #940: Release importer

Importing lookup values from the batch-index-lookup.json. Annotation types that matches a lookup in this file have their values converted to the mapped value. See also [4466].

comment:33 by Nicklas Nordborg, 7 years ago

(In [4486]) References #940: Release importer

Added support for producing a log file with information that may or may not be so useful.

comment:34 by Nicklas Nordborg, 7 years ago

(In [4487]) References #940: Release importer

The release import plug-in now ends with a "Done" message.

comment:35 by Nicklas Nordborg, 7 years ago

(In [4508]) References #940: Release importer

Updated opengrid.jar since it it has now been released (Open Grid 1.1).

comment:36 by Nicklas Nordborg, 7 years ago

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.