You are viewing limited content. For full access, please sign in.

Question

Question

Bulk deletes via Workflow without For Each

asked on December 11, 2024

Hey everyone

We are in a bit of a bind with Workflow and records maintenance and I'm hoping someone has a suggestion. Our vendor setup a business process that sends data to Workflow to start a query. This query identifies records to be removed due to them reaching retention limits. 

The issue is this is sometimes 20k-40k records per day. Workflow, it seems, has to repeat the query over and over for each record (for each) to delete each one. These workflows are backing up for days and causing general slowness across the org. 

We have a very busy environment, but we monitor resources very closely and our servers are never maxed out when this is happening... 

Maybe I'm misunderstanding the for each option in Workflow? Maybe there's a "delete entire results" vs a for each? Any help is appreciated. 

0 0

Answer

SELECTED ANSWER
replied on December 12, 2024

Large loops get slower as they go because they have to keep information about the previous iterations. We normally recommend that you keep iterations under 1000, around 500, if at all possible. You may get better performance if you spin the activities inside For Each into their own workflows so you can take advantage of Workflow's parallel execution abilities.

 

That said, is there a reason why you need to delete each document individually when you later seem to delete the parent folder? Is immediate deletion needed or would be ok if you run a workflow at the end of the month to just delete the folder? What are the criteria that govern the retention of these documents?

5 0

Replies

replied on December 11, 2024

It would be helpful if you could post one or more screenshots of the relevant part(s) of the workflow in question, including the properties of whichever activities are in the query invocation chain, especially the specifics of the query itself. If the "query" is a Search activity, provide the exact Advanced Search Syntax it uses.

Please make sure to redact any potentially sensitive information from the screenshots.

Re:

The issue is this is sometimes 20k-40k records per day. Workflow, it seems, has to repeat the query over and over for each record (for each) to delete each one. These workflows are backing up for days and causing general slowness across the org. 

We have a very busy environment, but we monitor resources very closely and our servers are never maxed out when this is happening... 

Most Workflow activities are not computationally intensive on the Workflow server itself. Workflow primarily sends requests off to other places (databases, repositories, etc.) and spends most of its time waiting for those requests to be fulfilled.

Finding and deleting 20-40k records per day is not something that should not be "backing up for days". There's a process design issue somewhere here.

2 0
replied on December 12, 2024

Thanks Samuel. I should have sent some screen shots to begin with. :) 

I've included screenshots of one of the workflows. This workflow finds receipt and check images at their retention dates, then removes them from the repository. This could be 20,000-50,000 daily. These take hours to complete. My understanding is the deletes are doing the "Find Checks" step, then the "For Each" is doing the delete. But does it redo the query after each individual delete? If not, I don't understand why these deletes would take so long.

For some additional context, this is one retention/purge workflow. There are others that run as well and more are to be built. I am just not understanding why simple deletes are taking so long to complete. 

Checks_Receipts1.PNG
Checks_Receipts2.PNG
0 0
SELECTED ANSWER
replied on December 12, 2024

Large loops get slower as they go because they have to keep information about the previous iterations. We normally recommend that you keep iterations under 1000, around 500, if at all possible. You may get better performance if you spin the activities inside For Each into their own workflows so you can take advantage of Workflow's parallel execution abilities.

 

That said, is there a reason why you need to delete each document individually when you later seem to delete the parent folder? Is immediate deletion needed or would be ok if you run a workflow at the end of the month to just delete the folder? What are the criteria that govern the retention of these documents?

5 0
replied on December 18, 2024

Hey @████████! Sorry for the late reply. I spoke with @████████ from CDI and he suggested the same as you. I am going to start redesigning our workflows to push to a "delete" workflow so I can run them in parallel. I didn't know this was possible. 

For some of the more simple ones, I may just try doing a directory remove as well. I appreciate your feedback! 

2 0
You are not allowed to follow up in this post.

Sign in to reply to this post.