Hello Jason, ya I just noticed that the only options are delete or move. I think I was thinking about QF agent. Doesn't that let you have no post-processing options? Anyways, my workflow was going to kick off on document creation so 1 workflow instance per document. I guess that might not be the most efficient. I wonder if doing 1 workflow that searches and processes the documents after they are imported would be better? idk. What do you think? 

So my first project with Laserfiche was to import about 45 million documents from a legacy system and I learned a lot of lessons in the process.

After some trial and error, and a lot of revisions early on, this is what we landed on:

1. Robo-copied all the source files, broke them into subfolders of about 200-250 files each, and gave the folders simple names like 00, 01, 02, etc. and grouped them in parent folders with the naming while trying to distribute everything as evenly as possible (i.e., 00\00,00\01,01\00,01\01)
2. I created an Import Agent profile that targeted the parent folder set to retrieve from subfolders, then used the folder\subfolder name as part of the repository path so the same structure would be maintained
3. I copied the IA profile to create one for each of the parent folders 
4. I set up a workflow (let's call it Subfolder WF) that looped through the contents of one subfolder (i.e., everything in 00\00) then set metadata, moved them, etc. and deleted the empty folder after the loop.
5. I set up another workflow (let's call it Folder WF) that would loop through all the contents of a parent folder. It would run Subfolder WF on the current entry of the loop and wait for it to finish
6. Finally, I set up a "Batch" workflow that would trigger the Folder WF on all of my "parent" folders simultaneously (either by looping and not waiting, or by using parallel branches) for controlled parallel processing.

The multiple IA profiles allowed me to pull things in faster, but be mindful of your IA server resources because more profiles only helps depending on how many threads the service can use at any given time.

The folders/workflows structured in that way allowed me to avoid running too many workflow instances while also avoiding the efficiency problems you get if you do too many loop iterations in a single workflow.

Running the parallel instances allowed me to speed things up, while also keeping tight control over how many could be running at once.

Now you do want to consider your WF activities when you decide how many items to loop through in a single subfolder. The more activities, the more data to track for each loop, and a workflow's execution can slow down dramatically if that builds up too much; it takes a lot, but in a bulk process like this it can definitely happen.

For example, at first I was looping through thousands of documents all in one go, then I realized it would start at about 7 sec per document but toward the end of the loop it would be closer to 1 min per document.

Breaking it into batches solved that problem because it broke it down into "bite size" pieces so I could finish a batch/instance before the data built up enough to affect performance time, and start a fresh one for the next batch.

gotcha, thank you for all the info. I'll do some more testing and I appreciate the info! The bite-size chunks does sound like a good idea because I know how WF can get bogged down. 

On that note, I'd create some sort of "holding" folder off the root, not put them in the root. That way, if anything goes wrong, your users don't have to wade through thousands of docs to get to their normal work.

If these documents need individual processing and there's no need to know anything from the other ones in the batch, the triggering on document creation IS the most efficient way to process them. That lets you take advantage of Workflow's multi-tasking capabilities (which you wouldn't if you got all folder contents and iterated over them). Processing is faster this way with less overhead on WF, LF and SQL servers.

I'll also note that Import Agent 10 throughput scales well with cores/vCPUs up to four, after which the marginal benefit of each additional core falls off hard. Two machines running Import Agent, each with 4 vCPU, will have something like 60-80% higher throughput than one machine with 8 vCPU.

Don't have Import Agent do page generation for the migration. It'll slow down your import throughput by an order of magnitude or more. If you need page gen, use DCC with Workflow on the backend.

@████████ Good points. I suppose I should rephrase. While triggered workflows for individual items is the most efficient from a performance standpoint, my concern would be on the database impact of creating so many instances strictly based on the problems we had with our bulk import.

This is obviously anecdotal and I don't think it affected processing times, but when we were triggering individual workflows with such a large number of documents it didn't take long before we started having issues with the instance history.

Again, I don't think it really affected throughput, but it did get to a point where we couldn't search workflow history reliably in the designer, and the automated database maintenance processes were failing (seemed to be timing out).

yeah, that makes sense Jason. I just ran a test and since I was using my production workflow server to process these docs, my users started complaining about slow performance with their normal use of LF. I had to stop importing and processing. I'm going to have to either limit what times the import is running or another idea I had was to spin up an extra workflow server just for processing these documents. The other thing I'm not sure how to accomplish is an extra IA instance. Even though I have RIO, I cannot install another IA program on an extra machine. I would love to do that to speed up importing based on what Samuel Carson suggested.  

I just tested importing about 12 GB of documents and after 17 hours, my system only made it through just over 10GB. Since I'm looking at 2-3 TB of data to import, if my math is correct, 5100 hours for 3TB or 212 days, and that would be running 24 hours a day. I definitely need to speed this up.