I might think of more to add later, but Sam covered a lot of good points; system resources and database maintenance will be crucial.

Our big repository is approaching 60 million document entries, each of which has at least 20 different metadata fields and it does indeed have a cumulative/exponential effect.

In addition to resource allocation, some other things to consider are folder structure, column displays in the client, update activities, and audit trail.

• To Sam's point about folder count estimates, in my experience, it is best to limit the number of entries within a folder for optimal client performance. For example, I started splitting things up into more subfolders because navigation can get very sluggish with 1,000 or more entries in a single folder (see next item).
• Additionally, the fewer columns displayed in the client the better, especially calculated values like Total Document Size; every column in the client has a cost and that adds up quick when you have a large database, and even more so when you have a lot in a single folder and/or a large number of search results.
• If the large size is accompanied by many frequent changes to entries, that can also be taxing on the workflow subscriber, and it can generate a massive amount of data in Audit Trail in addition to the transaction load on the database.

Another thing to be mindful of is volume size; logical volumes are a must, and our big repository is over 100TB at this point:

• We try limit each drive to no more than 4-8TB. The repository isn't really affected by logical volume size, but larger drives can cause headaches for backup and recovery processes.
• Leave a modest amount of breathing room on the drives; when you update a document with any new pages or electronic files, those will be added to that entry's current volume unless you migrate it, so you never want those older drives to be completely filled.

Basically, you want to think about the upfront implications like SQL performance and storage/resource optimization, but also make considerations for the expected level of insert/update activity, long-term growth, and implications for secondary systems that will interact with this repository.

We created a second repository where most of our document editing actually occurs to lessen the burden on the primary repository, i.e., 95% of the editing and workflow activity happens in a sort of 'staging' repository.

We only add/change things in the big one once a document is finalized in that first environment to minimize activity in the big repository; this is not practical in every environment, but I wanted to mention it since it provides important context for our resource allocation and our system load/performance.

UPDATE: Thanks for the correction Sam, 4GB would be a painfully small drive lol

; )

Hi @Samuel Carson and @Jason Smith,

Thanks for your replies. Tagging doesn't seem to work for me but hopefully you see this!

It's all great advice. Some of this was already part of my plan, some of this is new but having it all together in a single post, rocks.  🕺

Sam: The migrated data, which will be between 155 to 225 million records, will contain about 10 attributes each (to make sure we capture data from the priovious system). The next 75 to 145 million will contain around 5 atributes each. I'll be keeping my eye on the scope creep!

 Jason: 4 GB per volume, got it! They'll fit neatly on all the DVDs we have lying around 😜. We're lucky in that the folder structure isn't inteded to be be discovery-friendly so people will have to search, not browse. And at any rate, my filing workflow ensures no folder contains more that 1000 records. It also means I know the maximum number of folders up front. Also, the documents are edited elsewhere, before being stored in LF, so that'll help.

With your current system, do you know how many milliseconds it takes to retrieve a document using a known Entry ID, and by using Web Client or the .Net Framework API?

Regards,

Ben

Hi Ben,

I don't know the exact numbers, but by far the bigger factor in retrieval is the document size. Locating a document via Entry Id is nearly instant since those are indexed primary keys; the speed is more influenced by how long it takes to retrieve the pages and/or electronic file.