Mail notifications (e.g. "export metadata")

kerzel · 24 May 2023 10:04

Dear all,

some of the openBIS commands are asynchronous, e.g. “more” → “export Metadata”. Upon clicking. one gets a notification that a mail will be sent when the export is ready (and, I suppose, where to get it from).

However, unfortunately, the mail never arrives - I’m sure it’s “just” a configuration problem somewhere, but, tbh, I’m not sure where to look…

Mail in general works on the host where openBIS is installed, e.g. when I do

mail -s "test" <my email>
test text
crtrl-d

this test mail does arrive in my account soon after. I’ve added my email address to the user profile. openBIS itself can send mails as well, if I open the mail queue on the host it is running on, I can see a lot of mails like the following:

N6090 openbis@localhost  Wed May 24 12:00  23/914   ATTENTION: openBIS Server

where the body of the mail is essentially

2023-05-24 12:00:32,996 INFO  [cache-clearance-task - Maintenance Plugin] NOTIFY.CacheClearanceMaintenanceTask - Cache clearance execution finished.

(maybe it’s also not ideal to have many mails starting with ATTENTION and we need to fix another issue?)

Any suggestion where to look would be greatly appreciated…

Many thanks in advance
best wishes
Ulrich

cbarillari · 15 June 2023 14:02

Dear Ulrich,

this needs to be configured in the dss service properties. Installation and Administrators Guide of the openBIS Data Store Server - openBIS Documentation Rel. 20.10 - Confluence

kerzel · 16 June 2023 05:50

Dear Caterina,

many thanks. hm, maybe I miss the obvious - but could you give me a hint at the specific part of the documentation I need to look at? I think I may be stuck on a very basic level in this question that I can’t quite ask the right question…
My apologies for this rather newbie question
all the best
Ulrich

cbarillari · 16 June 2023 12:33

In

https://unlimited.ethz.ch/display/openBISDoc2010/Installation+and+Administrators+Guide+of+the+openBIS+Data+Store+Server#InstallationandAdministratorsGuideoftheopenBISDataStoreServer-Configurationfile

there is the following section:

SMTP properties (must start with ‘mail’ to be considered).
mail.smtp.host = localhost
mail.from = datastore_server@localhost
If this property is set a test e-mail will be sent to the specified address after DSS successfully started-up.
mail.test.address = test@localhost

kerzel · 26 June 2023 06:58

Dear Caterina

many thanks - we’ve updated the settings but I would guess the problem is (at least partially) elsewhere as well.
I can talk to Postgres and send an email if I open a session on port 25 - but not if I, for example, use “mail”…
What does openBIS do? Do you connect to the mail server directly, or do you use a helper program to send the mails?

All the best and many thanks
Ulrich

kerzel · 26 June 2023 15:45

Dear Caterina,

it seems we’re one step further- we can now do “export metadata” and receive and email with a download link. It also works if I do “export metadata and data” in the two cases: a) there is no data, b) there is only data that has been uploaded to openBIS.

Exporting data and metadata doesn’t work if I have a linked dataset (where the data themselves are on the S3 storage), if I delete the linked dataset, the export works again…?

Many thanks
Ulrich

cbarillari · 26 June 2023 15:51

Dear Ulrich,

it is expected that you cannot download data that are not directly stored in openBIS. However, what happens? Do you get an error that states that or the export simply fails?

kerzel · 27 June 2023 05:21

Dear Caterina,

looking at the logfile, the process crashes.
I’ve pasted the dictionaries for the linked data sent to openBIS below, the linked file seems to work (although, of course, I may be doing something wrong). It appears like this in the ELN UI

The external location was created via

dms_test_001 = oBis.create_external_data_management_system(code='S3_test_001', 
                                                           label='TestBucket01 in CoScine', 
                                                           address='COSCINE:/RWTH-RDS-S3/TestBucket01',
                                                           address_type='FILE_SYSTEM' )

(it is, indeed, not exactly a file system but I thought it was the closest match from the list of options available - however, looking at the error message below it may not be?)

Then when I try to do export data and metadata, the process crashes with the following (more of the traceback below)

2023-06-27 07:05:08,821 INFO  [Thread-60] ACCESS.DataStoreServerApiLogger - [SESSION:ulrichkerzel-230627065141832x8FF44A9416B0BB7170E39E2B422E660B]: (16ms) download-files  FILE_IDS('[DataSetFilePermId[dataSetId=20230627070442527-474,filePath=COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png]]') DOWNLOAD_OPTIONS('DataSetFileDownloadOptions[recursive=false]')
2023-06-27 07:05:08,823 ERROR [Thread-60] OPERATION.generalExports.py - Error occurred: java.lang.IllegalArgumentException: Resource 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png' is currently unavailable. It might be archived.
java.lang.IllegalArgumentException: Resource 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png' is currently unavailable. It might be archived.
        at ch.systemsx.cisd.openbis.dss.generic.shared.content.PathInfoProviderBasedHierarchicalContent.getFileContentProvider(PathInfoProviderBasedHierarchicalContent.java:370)
        at ch.systemsx.cisd.openbis.dss.generic.shared.content.PathInfoProviderBasedHierarchicalContent.access$200(PathInfoProviderBasedHierarchicalContent.java:46)

Looking at the documentation for Link Data Sets, one can have: a remote openBIS instance, a file system, or an URL
where I thought that file system would be the closest match for our use-case (data on an externally managed S3).
However, any of those might be offline anyway?

For linked data, I would be happy enough if the data export would return the details to where on the external store the linked data can be found.

Many thanks again for all the help
Ulrich

Dictionaries for linked file with debug info

DEBUG:root:openBIS link metadata dict: {'linkedData': {'@type': 'as.dto.dataset.create.LinkedDataCreation', 'contentCopies': [{'@type': 'as.dto.dataset.create.ContentCopyCreation', 'path': 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png', 'externalDmsId': {'permId': 'S3_TEST_001', '@type': 'as.dto.externaldms.id.ExternalDmsPermId'}}]}, 'typeId': {'@type': 'as.dto.entitytype.id.EntityTypePermId', 'permId': 'RAW_DATA'}, 'dataStoreId': {'permId': 'DSS1', '@type': 'as.dto.datastore.id.DataStorePermId'}, 'parentIds': [], 'measured': False, 'properties': {'$NAME': 'DemoLinkedData_002'}, '@type': 'as.dto.dataset.create.DataSetCreation', 'sampleId': {'identifier': '/ULRICHKERZEL/TRAINING_PROJECT_001/MEASURE_POINT369', '@type': 'as.dto.sample.id.SampleIdentifier'}, 'autoGeneratedCode': True}
DEBUG:root:openBIS link metadata:
 {'linkedData': {'@type': 'as.dto.dataset.create.LinkedDataCreation', 'contentCopies': [{'@type': 'as.dto.dataset.create.ContentCopyCreation', 'path': 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png', 'externalDmsId': {'permId': 'S3_TEST_001', '@type': 'as.dto.externaldms.id.ExternalDmsPermId'}}]}, 'typeId': {'@type': 'as.dto.entitytype.id.EntityTypePermId', 'permId': 'RAW_DATA'}, 'dataStoreId': {'permId': 'DSS1', '@type': 'as.dto.datastore.id.DataStorePermId'}, 'parentIds': [], 'measured': False, 'properties': {'$NAME': 'DemoLinkedData_002'}, '@type': 'as.dto.dataset.create.DataSetCreation', 'sampleId': {'identifier': '/ULRICHKERZEL/TRAINING_PROJECT_001/MEASURE_POINT369', '@type': 'as.dto.sample.id.SampleIdentifier'}, 'autoGeneratedCode': True}
DEBUG:root:openBIS link request: {'method': 'createDataSets', 'params': ['ulrichkerzel-230627065141832x8FF44A9416B0BB7170E39E2B422E660B', [{'fileMetadata': [{'fileLength': 450301, 'crc32': 0, 'crc32Checksum': 0, 'checksum': '7aa8166a8def7a21', 'checksumType': 'xxHash', 'directory': False, 'path': 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png'}], 'metadataCreation': {'linkedData': {'@type': 'as.dto.dataset.create.LinkedDataCreation', 'contentCopies': [{'@type': 'as.dto.dataset.create.ContentCopyCreation', 'path': 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png', 'externalDmsId': {'permId': 'S3_TEST_001', '@type': 'as.dto.externaldms.id.ExternalDmsPermId'}}]}, 'typeId': {'@type': 'as.dto.entitytype.id.EntityTypePermId', 'permId': 'RAW_DATA'}, 'dataStoreId': {'permId': 'DSS1', '@type': 'as.dto.datastore.id.DataStorePermId'}, 'parentIds': [], 'measured': False, 'properties': {'$NAME': 'DemoLinkedData_002'}, '@type': 'as.dto.dataset.create.DataSetCreation', 'sampleId': {'identifier': '/ULRICHKERZEL/TRAINING_PROJECT_001/MEASURE_POINT369', '@type': 'as.dto.sample.id.SampleIdentifier'}, 'autoGeneratedCode': True}, '@type': 'dss.dto.dataset.create.FullDataSetCreation'}]]}
DEBUG:root:openBIS link request
 {'method': 'createDataSets', 'params': ['ulrichkerzel-230627065141832x8FF44A9416B0BB7170E39E2B422E660B', [{'fileMetadata': [{'fileLength': 450301, 'crc32': 0, 'crc32Checksum': 0, 'checksum': '7aa8166a8def7a21', 'checksumType': 'xxHash', 'directory': False, 'path': 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png'}], 'metadataCreation': {'linkedData': {'@type': 'as.dto.dataset.create.LinkedDataCreation', 'contentCopies': [{'@type': 'as.dto.dataset.create.ContentCopyCreation', 'path': 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png', 'externalDmsId': {'permId': 'S3_TEST_001', '@type': 'as.dto.externaldms.id.ExternalDmsPermId'}}]}, 'typeId': {'@type': 'as.dto.entitytype.id.EntityTypePermId', 'permId': 'RAW_DATA'}, 'dataStoreId': {'permId': 'DSS1', '@type': 'as.dto.datastore.id.DataStorePermId'}, 'parentIds': [], 'measured': False, 'properties': {'$NAME': 'DemoLinkedData_002'}, '@type': 'as.dto.dataset.create.DataSetCreation', 'sampleId': {'identifier': '/ULRICHKERZEL/TRAINING_PROJECT_001/MEASURE_POINT369', '@type': 'as.dto.sample.id.SampleIdentifier'}, 'autoGeneratedCode': True}, '@type': 'dss.dto.dataset.create.FullDataSetCreation'}]]}
DEBUG:root:DSS URL: https://openbis.imm.rwth-aachen.de:443/datastore_server/rmi-data-store-server-v3.json
DEBUG:root:URL for the link request (DSS): https://openbis.imm.rwth-aachen.de:443/datastore_server/rmi-data-store-server-v3.json
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openbis.imm.rwth-aachen.de:443
DEBUG:urllib3.connectionpool:https://openbis.imm.rwth-aachen.de:443 "POST /datastore_server/rmi-data-store-server-v3.json HTTP/1.1" 200 123
DEBUG:root:Server response to POST command: [{'@type': 'as.dto.dataset.id.DataSetPermId', '@id': 1, 'permId': '20230627070442527-474'}]
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openbis.imm.rwth-aachen.de:443
DEBUG:urllib3.connectionpool:https://openbis.imm.rwth-aachen.de:443 "POST /openbis/openbis/rmi-application-server-v3.json HTTP/1.1" 200 1942
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openbis.imm.rwth-aachen.de:443
DEBUG:urllib3.connectionpool:https://openbis.imm.rwth-aachen.de:443 "POST /openbis/openbis/rmi-application-server-v3.json HTTP/1.1" 200 885

DSS Traceback*

2023-06-27 07:05:08,803 INFO  [Thread-60] OPERATION.exportsApi.py - --> Entity type: FILE permId: 20230626172850663-472 obj: False path: None before files.
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - exporting type: FILE permId: 20230627070442527-474
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - --> Entity type: FILE permId: 20230627070442527-474 obj: False path: None before files.
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - exporting type: FILE permId: 20230627070442527-474
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - --> Entity type: FILE permId: 20230627070442527-474 obj: False path: None before files.
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - exporting type: FILE permId: 20230627070442527-474
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - --> Entity type: FILE permId: 20230627070442527-474 obj: False path: None before files.
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - exporting type: FILE permId: 20230627070442527-474
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - --> Entity type: FILE permId: 20230627070442527-474 obj: False path: None before files.
2023-06-27 07:05:08,804 INFO  [Thread-60] OPERATION.exportsApi.py - exporting type: FILE permId: 20230627070442527-474
2023-06-27 07:05:08,821 INFO  [Thread-60] ACCESS.DataStoreServerApiLogger - [SESSION:ulrichkerzel-230627065141832x8FF44A9416B0BB7170E39E2B422E660B]: (16ms) download-files  FILE_IDS('[DataSetFilePermId[dataSetId=20230627070442527-474,filePath=COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png]]') DOWNLOAD_OPTIONS('DataSetFileDownloadOptions[recursive=false]')
2023-06-27 07:05:08,823 ERROR [Thread-60] OPERATION.generalExports.py - Error occurred: java.lang.IllegalArgumentException: Resource 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png' is currently unavailable. It might be archived.
java.lang.IllegalArgumentException: Resource 'COSCINE:/RWTH-RDS-S3/TestBucket01/mini_sem.png' is currently unavailable. It might be archived.
        at ch.systemsx.cisd.openbis.dss.generic.shared.content.PathInfoProviderBasedHierarchicalContent.getFileContentProvider(PathInfoProviderBasedHierarchicalContent.java:370)
        at ch.systemsx.cisd.openbis.dss.generic.shared.content.PathInfoProviderBasedHierarchicalContent.access$200(PathInfoProviderBasedHierarchicalContent.java:46)
        at ch.systemsx.cisd.openbis.dss.generic.shared.content.PathInfoProviderBasedHierarchicalContent$PathInfoNode.getContentProvider(PathInfoProviderBasedHierarchicalContent.java:330)
        at ch.systemsx.cisd.openbis.dss.generic.shared.content.PathInfoProviderBasedHierarchicalContent$PathInfoNode.doGetInputStream(PathInfoProviderBasedHierarchicalContent.java:322)
        at ch.systemsx.cisd.openbis.common.io.hierarchical_content.AbstractHierarchicalContentNode.getInputStream(AbstractHierarchicalContentNode.java:142)
        at ch.systemsx.cisd.openbis.common.io.ConcatenatedContentInputStream.createContentStream(ConcatenatedContentInputStream.java:210)
        at ch.systemsx.cisd.openbis.common.io.ConcatenatedContentInputStream.read(ConcatenatedContentInputStream.java:124)
        at java.base/java.io.InputStream.read(InputStream.java:271)
        at ch.ethz.sis.openbis.generic.dssapi.v3.dto.datasetfile.download.DataSetFileDownloadReader$1.read(DataSetFileDownloadReader.java:86)
        at java.base/java.io.InputStream.read(InputStream.java:205)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1025)
        at jdk.internal.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.python27.core.PyReflectedFunction.__call__(PyReflectedFunction.java:190)
        at org.python27.core.PyReflectedFunction.__call__(PyReflectedFunction.java:208)
        at org.python27.core.PyObject.__call__(PyObject.java:477)
        at org.python27.core.PyObject.__call__(PyObject.java:481)
        at exportsApi$py.generateFilesInZip$12(/home/openbis/openbis/servers/datastore_server/../core-plugins/eln-lims/10/dss/reporting-plugins/exports-api/exportsApi.py:409)
        at exportsApi$py.call_function(/home/openbis/openbis/servers/datastore_server/../core-plugins/eln-lims/10/dss/reporting-plugins/exports-api/exportsApi.py)
        at org.python27.core.PyTableCode.call(PyTableCode.java:173)
        at org.python27.core.PyBaseCode.call(PyBaseCode.java:306)
      at org.python27.core.PyFunction.function___call__(PyFunction.java:474)
        at org.python27.core.PyFunction.__call__(PyFunction.java:469)
        at org.python27.core.PyFunction.__call__(PyFunction.java:459)
        at org.python27.pycode._pyx307.generateZipFile$5(<string>:108)
        at org.python27.pycode._pyx307.call_function(<string>)
        at org.python27.core.PyTableCode.call(PyTableCode.java:173)
        at org.python27.core.PyBaseCode.call(PyBaseCode.java:306)
        at org.python27.core.PyFunction.function___call__(PyFunction.java:474)
        at org.python27.core.PyFunction.__call__(PyFunction.java:469)
        at org.python27.core.PyFunction.__call__(PyFunction.java:459)
        at org.python27.pycode._pyx307.export$3(<string>:86)
        at org.python27.pycode._pyx307.call_function(<string>)
        at org.python27.core.PyTableCode.call(PyTableCode.java:173)
        at org.python27.core.PyBaseCode.call(PyBaseCode.java:306)
        at org.python27.core.PyFunction.function___call__(PyFunction.java:474)
        at org.python27.core.PyFunction.__call__(PyFunction.java:469)
        at org.python27.core.PyFunction.__call__(PyFunction.java:464)
        at org.python27.core.PyObject._callextra(PyObject.java:589)
        at threading$py.run$35(threading.py:213)
        at threading$py.call_function(threading.py)
        at org.python27.core.PyTableCode.call(PyTableCode.java:173)
        at org.python27.core.PyBaseCode.call(PyBaseCode.java:134)
        at org.python27.core.PyFunction.__call__(PyFunction.java:416)
        at org.python27.core.PyMethod.__call__(PyMethod.java:126)
        at threading$py._Thread__bootstrap$36(threading.py:261)
        at threading$py.call_function(threading.py)
        at org.python27.core.PyTableCode.call(PyTableCode.java:173)
        at org.python27.core.PyBaseCode.call(PyBaseCode.java:306)
        at org.python27.core.PyBaseCode.call(PyBaseCode.java:197)
        at org.python27.core.PyFunction.__call__(PyFunction.java:485)
        at org.python27.core.PyMethod.instancemethod___call__(PyMethod.java:237)
        at org.python27.core.PyMethod.__call__(PyMethod.java:228)
        at org.python27.core.PyMethod.__call__(PyMethod.java:218)
        at org.python27.core.PyMethod.__call__(PyMethod.java:213)
        at org.python27.core.FunctionThread.run(FunctionThread.java:23)

juanf · 28 June 2023 12:46

Dear Ulrich,

I can imagine is very annoying finding out that exports fail if link datasets are found.

I think on this case exports should ignore the link datasets data and just include the metadata.

We should fix this soon. Maybe it can make it to the next minor release 20.10.7

Thanks for reporting this,
Juan

kerzel · 28 June 2023 14:10

Dear @juanf ,

Great, thank you very much.

All the best
Ulrich