Opened 18 months ago

Closed 18 months ago

#1488 closed enhancement (fixed)

Auto-import JSON files

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: major Milestone: Reggie v4.48
Component: net.sf.basedb.reggie Keywords:
Cc:

Description

The flow of JSON files is working pretty well at the moment. It should be possible to implement an "auto-import" functionality for all JSON files that are verified without errors or warnings.

Change History (8)

comment:1 by Nicklas Nordborg, 18 months ago

In 7221:

References #1488: Auto-import JSON files

Started to implement an auto-importer for JSON files. It can search for and validate JSON files, but the actual import is not yet implemented. It would be nice to keep some information as annotations on the FileServer item. This requires an update to BASE. See ​https://base.thep.lu.se/ticket/2305

comment:2 by Nicklas Nordborg, 18 months ago

In 7223:

References #1488: Auto-import JSON files

Added support for the AutoProcessing annotation in the import gateway. If it is set to "Disable" no automatic import will be done.

comment:3 by Nicklas Nordborg, 18 months ago

In 7224:

References #1488: Auto-import JSON files

Added FilesToIgnore annotation so that we can keep track of JSON files with errors, warnings or other problems and don't have to try importing them the next time. Each JSON file will have at most one chance for the import to succeed.

comment:4 by Nicklas Nordborg, 18 months ago

In 7225:

References #1488: Auto-import JSON files

Implemented a way to specify which user account that should be used for the auto-import. The same account is also used for the rest of the secondary analysis pipeline.

The user account is specified by creating an any-to-any link named named "AutoImportAccount" on the gateway file-server item that points to the desired account.

comment:5 by Nicklas Nordborg, 18 months ago

In 7226:

References #1488: Auto-import JSON files

Implemented the actual import of the JSON file. All items seems to be created as expected.

comment:6 by Nicklas Nordborg, 18 months ago

In 7227:

References #1488: Auto-import JSON files

Enabled the ExternalOperator annotation for file servers. We can set this on the gateway item to use for the import.

comment:7 by Nicklas Nordborg, 18 months ago

In 7228:

References #1488: Auto-import JSON files

Implemented support for sending FASTQ import jobs to the cluster once the JSON import is completed.

comment:8 by Nicklas Nordborg, 18 months ago

Resolution: → fixed
Status: new → closed
Note: See TracTickets for help on using tickets.