Having large holds or many holds shouldn't cause a performance problem by itself. Each entry under hold is represented by a simple relation row in a database table, so even a million entries in a hold is not very much data. But it is possible that your system was at its limit just before this. You'll want to take a look at the SQL Server first, and make sure that has the resources it needs.

It's also possible that something else changed around the same time, and the holds turn out to be a red herring. Especially since you mention that opening a document is slow. When you open a document, I don't think it fetches the hold list unless the user explicitly chooses to do that. Until then, it just checks if there is a single applicable hold, which should be fast. Is there a performance problem interacting with a document that is not a part of any hold?

When you say that it's "crashing when processing holds", can you clarify what action you are taking in the application, and what you see when it doesn't work? If there is an error shown to the user, often there is additional information in the event logs on the servers.