Opened 5 months ago

Closed 9 days ago

#940 closed task (fixed)

Release importer

Reported by: nicklas Owned by: nicklas
Priority: critical Milestone: Relax v1.0
Component: net.sf.basedb.relax Keywords:
Cc:

Description

A release importer plug-in or wizard needs to be implemented. It should be able to import a release generated from the Reggie Release exporter plug-in. It should look for the files on a FileServer item.

Change History (36)

comment:1 Changed 4 months ago by nicklas

  • Owner set to nicklas
  • Status changed from new to assigned

comment:2 Changed 4 months ago by nicklas

(In [4366]) References #940: Release importer

Started to work on a release importer plug-in. It connects to a selected FileServer item (we use the generic SSH functionality in the Open Grid extension) and looks for files that the exporter (in Reggie) is supposed to create. Currently it will only count the number of *.json files and check for the exportcomplete file. Once found the plug-in exits.

comment:3 Changed 4 months ago by nicklas

(In [4368]) References #940: Release importer

The release import plug-in now load and parse all JSON files and create a few items (samples and extracts). So far, only the name and project key properties are set.

This code should be considered proof-of-concept only, and need a lot of re-factoring before it is usable in practice.

comment:4 Changed 4 months ago by nicklas

(In [4370]) References #940: Release importer

The release importer now set the subtype and parent items (for samples and extracts).

comment:5 Changed 4 months ago by nicklas

(In [4372]) References #940: Release importer

Started to refactor the release importer so that each item type is handled by it's own class. It feels a bit outdated already, but it should work with the current exporter.

comment:6 Changed 4 months ago by nicklas

(In [4381]) References #940: Release importer

Added support for importing annotation types. Annotation types are created and shared to the active project in the same manner as cohort items. This means that annotation types are only visible in projects(=releases) that they are part of. The project-specific annotation flag is not yet handled.

Parts of the code related to the remote file server has been temporarily disabled or modified to make the plug-in work with files that are located on local file system instead due to not having (easy) access to a real file server at the moment.

comment:7 Changed 4 months ago by nicklas

(In [4383]) References #940: Release importer

Now reads the files.json (see [4382]) to get the list of cohort data files.

comment:8 Changed 4 months ago by nicklas

(In [4384]) References #940: Release importer

Added support for importing annotation values.

Switched to BASE 3.11 since we need the new API for handling project annotations.

comment:9 Changed 4 months ago by nicklas

(In [4386]) References #940: Release importer

Started to clean up the code. Error handling has been improved. The code should now be ready for the more advanced stuff that need to be imported.

comment:10 Changed 4 months ago by nicklas

(In [4387]) References #940: Release importer

The plug-in now checks that a project is active before it is used and validates that the "Export ID" parameter only contains valid characters.

comment:11 Changed 4 months ago by nicklas

(In [4388]) References #940: Release importer

Moved some of the annotation type importer code.

comment:12 Changed 4 months ago by nicklas

(In [4390]) References #940: Release importer

Support for setting the project-specific annotations flag when importing annotations.

comment:13 Changed 4 months ago by nicklas

(In [4391]) References #940: Release importer

Revert back to using real file server and SSH again.

comment:14 Changed 4 months ago by nicklas

(In [4393]) References #940: Release importer

More cleanup.

comment:15 Changed 4 months ago by nicklas

(In [4395]) References #940: Release importer

Importing the registration date.

comment:16 Changed 4 months ago by nicklas

(In [4397]) References #940: Release importer

Importing the creation date.

comment:17 Changed 3 months ago by nicklas

(In [4399]) References #940: Release importer

Added MergedSequences subtype.

comment:18 Changed 3 months ago by nicklas

(In [4400]) References #940: Release importer

Added MergedSequences subtype.

comment:19 Changed 3 months ago by nicklas

(In [4403]) References #940: Release importer

Importing the DataFilesFolder annotation. We need special handling for this annotation. When creating a new item the default value is always set to the original path. Project-specific values are always created using the release version as a top-level prefix.

For example:

Default value: /S000006/l.r.lib.g/k.a/t
Release 0.9:   /0.9/S000006/l.r.lib.g/k.a/t

comment:20 Changed 3 months ago by nicklas

(In [4405]) References #940: Release importer

Importing and creating files in the BASE file system. The files are not yet linked to the owning item.

comment:21 Changed 3 months ago by nicklas

(In [4411]) References #940: Release importer

Importing from the new typedefs.json file. Support for importing data file types. Linking files to items.

comment:22 Changed 3 months ago by nicklas

(In [4413]) References #940: Release importer

Importing files that are linked with any-to-any links.

comment:23 Changed 3 months ago by nicklas

(In [4429]) References #940: Release importer

The release version number is now used for importing instead of the UUID.

comment:24 Changed 3 months ago by nicklas

(In [4433]) References #940: Release importer

Removed debug code.

Default value for file server is set to the ReleaseArchive server.

Default value for release version is extracted from the project name (if it contains a version-like pattern).

comment:25 Changed 3 months ago by nicklas

(In [4436]) References #940: Release importer

Raw bioassays are created with the raw data type, platform or platform variant specified in the exported file.

comment:26 Changed 3 months ago by nicklas

(In [4440]) References #940: Release importer

Added support for resuming the import after a system restart.

comment:27 Changed 3 months ago by nicklas

(In [4441]) References #940: Release importer

Added some counters used for a final summary message.

comment:28 Changed 3 months ago by nicklas

(In [4445]) References #940: Release importer

Added support for pausing the import.

comment:29 Changed 3 months ago by nicklas

(In [4446]) References #940: Release importer

The importer now has support for adding annotations to existing items even if the annotation type is not marked as project-specific. The use case is that the list of annotation types has changed between two releases. Annotations that are new in the later relase are allowed to be set on already existing items. This will not affect the previous release since those annotations are not visible in that release (except if the logged in user is an admin).

In this sense we consider an annotation type to be "new" if it (after importing it) is only shared to the current project.

comment:30 Changed 3 months ago by nicklas

(In [4447]) References #940: Release importer

Resuming after a pause should now be implemented in a more stable manner. There was a risk that the same file was processed twice since there was a rather large gap between the transaction commit and the change of the breakpoint to the next file.

Re-configuration after a pause now checks if the relase version parameter has changed and forces a restart from the beginning in that case since the stored state information is no longer valid.

comment:31 Changed 2 months ago by nicklas

(In [4463]) References #940: Release importer

Added support for importing annotation types with units.

comment:32 Changed 2 months ago by nicklas

(In [4467]) References #940: Release importer

Importing lookup values from the batch-index-lookup.json. Annotation types that matches a lookup in this file have their values converted to the mapped value. See also [4466].

comment:33 Changed 7 weeks ago by nicklas

(In [4486]) References #940: Release importer

Added support for producing a log file with information that may or may not be so useful.

comment:34 Changed 7 weeks ago by nicklas

(In [4487]) References #940: Release importer

The release import plug-in now ends with a "Done" message.

comment:35 Changed 7 weeks ago by nicklas

(In [4508]) References #940: Release importer

Updated opengrid.jar since it it has now been released (Open Grid 1.1).

comment:36 Changed 9 days ago by nicklas

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.