Uploading and uncompressing zipfile using pybis

Developments
1

Hi all,

I am trying to upload a zip file as dataset using pybis==1.37.4 (Testen on OpenBis versions 20.10.11.1 and 20.10.12.5).

In general the file upload is working, however I am trying to set the option to uncompress the file in openbis so that all individual files of the zip archive show in the dataset (instead of just archive.zip)

Here is my example code (assuming openbis connection o and entring some permID for the file to attach the dataset):

# Source - https://stackoverflow.com/a/44946732
# Posted by Vladimir, modified by community. See post 'Timeline' for change history
# Retrieved 2026-03-03, License - CC BY-SA 4.0

import io
import zipfile

filename = "./archive.zip"

# create zip archive
zip_buffer = io.BytesIO()
with zipfile.ZipFile(zip_buffer, "a", zipfile.ZIP_DEFLATED, False) as zip_file:
    for file_name, data in [
        ("1.txt", io.BytesIO(b"111")),
        ("2.txt", io.BytesIO(b"222")),
    ]:
        zip_file.writestr(file_name, data.getvalue())

# write to disk
with open(filename, "wb") as f:
    f.write(zip_buffer.getvalue())

# upload to dataset with pybis
object_permid = "" # set some permId for testing here
obj = o.get_object(object_permid)
dataset = o.new_dataset(type="ATTACHMENT", sample=obj, zipfile=filename)
dataset.save()

If I upload the file in the web UI and select the “uncompress before import” option everything works as expected

image
image225×161 3.81 KB

image

With pybis I end up with this instead:

image

The relevant section in the pybis documentation seems to use the same command syntax

ds_new = o.new_dataset(
    type       = 'RAW_DATA',
    sample     = '/SPACE/SAMP1',
    zipfile    = 'my_zipped_folder.zip',
)
ds_new.save()

Any hints on what I am doing wrong?

thanks in advance,
Cagtay

2

Hi Cagtay,

It might not be possible through pybis.

pybis uses API v3 if you have the newer version of openBIS

You can extract the contents of the zip file into a directory and provide the path to the directory when you create the new dataset. This behaves the same as with the GUI (with the unpack option on).

Hope this helps!

3

Thank you for responding @khalil.rejiba

I didn’t know there was a difference regarding this functionality when using API V3.
It is still listen in the documentation even though that mentions API V3 explicitly as well.

docs/software-developer-documentation/apis/python-v3-api.md · master · sispub / openbis · GitLab

Maybe someone from the openBIS team can clarify of this option is still supported.

Of course decompressing to a directory on disk is another option, I might have to go that route. I was trying to avoid all the disc access overhead since the zip contents are generated entirely in memory and compress down nicely to around 85% for >50 files :pensive_face: