Question

Efficient Volume Copying

Version 10 Administration

Updated July 29, 2021

asked on July 29, 2021

We are trying to copy a few of our production volumes into Dev to test an upgrade. The size is between 2-5 TB worth of data. Is there a more efficient way to copy these from one server to another? Right now I am using robocopy to move a volume that is about 255 GB and it has been running for about 2 hrs without completion. At this rate I am afraid we will never get started due to waiting for the volumes to be copied.

Any suggestions or help is greatly appreciated

0 0

Replies

replied on July 29, 2021

2-5TB? A Terrabyte is 1,000,000,000,000 Bytes

If your using a 1Gb/s network connection, that is a little over 100MB/s transfer speed. Also a disk of that size might only be 100MB/s write speed even if your network connection is faster.

There are only 3,600 seconds in an hour but at 100MB/s you would need 10,000 seconds to write/transfer only 1TB.

You might be looking at days unless you have crazy fast hardware. You might need a fiber network connection and 550MB/s solid state storage to move that kind of data within hours.

1 0

replied on July 29, 2021

Chad basically has the math right. The millions of tiny files in Laserfiche volumes also incur tons of overhead with robocopy (and any other file copying utilities) so you'll often hit bottlenecks there too.

First, ask yourself if you really need 2-5 TB worth of volume data to test this upgrade. You almost certainly do not. Consider that tests you're planning to run in Dev and think about which test cases are actually dependent on successfully retrieving a volume file. Non-prod repositories with only a small subset of entries actually having images/files behind them are quite common for this reason.

If you do really need all that data copied, in your virtualization/storage platform see if there's any way to take a snapshot of the Prod volume data disk and restore a copy of it attached to Dev. That operation, if possible, will be much faster than robocopy because it works at the block level instead of the file level and thus avoid a significant amount of performance overhead.

1 0

replied on July 29, 2021

Thanks for the response Sam, also good to hear from you, in the time since I posted this I was given a more efficient method, in the same ballpark as your suggestion, to get this done so we are preparing to do that. Unfortunately, the business unit does want to completely duplicate the PROD environment to test the upgrade. So we are trying to make that accommodation for them

0 0

replied on July 29, 2021

I can't say how things work for sure, but I am almost certain that when you start the Laserfiche Service, it doesn't even touch the raw files. Only when you open a specific file from the repository does it access a raw file in any way. The reason I think this is because there is never an error or problem when raw files are missing.

This means if you start the updated service, even with all the files copied, you have not confirmed anything in regards to accessing any of those files in the new version. Any file not opened would not be tested in any way, since it was never touched.

Only Laserfiche can confirm this of course, it is just speculation.

0 0

replied on July 29, 2021

This is correct to the best of my knowledge. Laserfiche Server upgrades for any version 8+ do not touch/modify volume files, only the database.

Volume files you're not going to actually open provide no useful information for testing an upgrade because LFS will never interact with them.

Good to hear from you too LJ =)

2 0

You are not allowed to follow up in this post.

Question

Question

Efficient Volume Copying

Replies

Sign in to reply to this post.