Custom Data archive and/or export?

Data Management
1

Dear all,

I was wondering if it would be possible to customise and/or add new plug-ins for exporting and archving data. Our university develops and hosts a platform called CoScine that is also intended to share and collaboarate with data. One use-case I could foresee would be that researchers distributed across multiple institutions can use this to share data (and then the final export can be made to zenodo).
Sharing data within openBIS is possible as well if I understood things correctly - but then the data would have to be downloaded for each further use. The Coscine facilities are linked to an S3 object store that is also accessible from the cluster environment, etc - essentially, the researchers could, if possible, export the data they want to share there and it can then be used by others withouth having to copy data across.

Similarly, would it be possible to add an export option that uses tar instead of zip (or an option to choose)? We’re also trying to find ways to circument the “small file problem” for machine learning, WebDataset (https://webdataset.github.io) seems a promising approach. We could, of course, download the data from openBIS, create the archives, upload this to a different store, register the meta-data in openBis - but I was wondering if this process could be made easier for the user and some steps integrated in openBIS.
I understand, of course, that such a functionality would have to be developed - if we had (our could find) ressources for someone to help, how feasible would it be to do so?

Many thanks
all the best
Ulrich

2

Dear Ulrich,

Using a format like tar instead of zip would be a welcome change/option. Should align better for long term preservation archives.

Regarding adding export options. Is possible already to:

  • Extend the ELN UI adding custom views.
  • Extend the backend providing your own AS / DSS service with a custom API.

So you should be able to develop your own utilities without changing the core.

I’m sometimes surprised by some users integrations that modify the ELN in unexpected ways, sometimes with more hacks that they should. If you would like to extend current forms toolbars, header, etc… on the ELN let us know since it would be very straightforward to include those in the current interceptor pattern used by the ELN plugin interface, that would be the safe and clean approach.

Best,
Juan

3

Dear Juan,

great, many thanks, that sounds very good.
We would indeed aim to contribute something that can also be used in the community rather than hacking something together - we’re still in very early stages with our openBIS journey. It would be then great to reach out at some point in the not too far future how we could go about such developments best (we may even be able to allocate dedicated ressources)

All the best
Ulrich