How fast is Excel batch registration?

ELN
1

Hello.

I would like to know how long does it take to import approximately 12,000 samples together with their storage positions (so 24,000 objects) via the Excel batch registration?

I am now waiting for more than 12 hours. Is this normal or not?

We allready batch registrated 60,000 chemicals via one Excel file. It took maybe 2-3 hours. However, without storage positions. Is this the problem?

Viktoria

2

Hi Viktoria,

What openBIS version are you running? How much ram are you dedicating to AS and Postgres? Can you share with me your data model and batch upload so I can try it? Would love to try your use case hands on.

We introduced may performance improvements on the XLS batch registration.

I would expect 1000 objects get done in about a couple minutes. But if you are creating huge transactions you might need to tune both AS and Postgres memory requirements accordingly.

Best,
Juan

1 Like
3

Dear Juan.

Current Version: 20.10.7.2
Ram to AS: For Java 2GB and for caching 8GB
Ram to postgres 128MB

We are using the Object Type “OLIGO”.
11808 Oligos with corresponding 11808 storage positions (one storage position per oligo).
No other parents or children.

We recognized that for sample registration many core CPUs are used, but after that the software is doing something with storage positions we do not understand and for this only one of our 16 core CPUs is used with 100%.

Up to now we did not manage to upload the list. We tryed for the first time last week.
The same Excel file with a reduced number of objects (100) was uploaded within few minutes.

thanks
Viktoria

4

We have now managed to upload 5000 oligos, but it took 19 hours.

5

Dear Juan.

To keep you up to date:
As I mentioned earlier, the first 5000 oligos with the corresponding storage positions took about 19 hours.
For the remaining 6810 Oligos with storage positions, it took about 48 hours.
It’s not proportionate.

Overall, it takes too long. How could we speed it up? Did you have time to try our case by yourself?

Viktoria

6

Hi Viktoria,

Can you share with me your data model and batch upload so I can try it? Would love to try your use case hands on to see what is going on.

Just open a ticket and attach them please: openbis-support@id.ethz.ch

Please mention me on the ticket so they redirect the ticket to me directly.

Best,
Juan