Run the statistics report to see what you have in there: from the Workflow Designer, open the Workflow menu and choose Create Workflow Statistics Report. You don't have to select any filters. This should give you an idea of the numbers of instances running for each workflow definition.

If any of the numbers looks higher than expected for any of them, then you might have a faulty starting rule.

If the numbers look as expected, you may want to tweak the number of days completed workflow history is kept in the database. By default, that's 30 days. The value can be updated from the Workflow administration console, under Advance Server Options.

Other possible causes for the size of the database can be excessive use of Track Tokens or inefficient starting rules.