So when using forms i'm not aware of anyway to limit who can be reassigned to, but you have a few options on how to handle this.

1) If you disable the ability to reassign forms, if you are using teams they should be able to release the task back into the pool for someone else in the team to pick up. Depending on how you are assigning them ofc.

2)Create your own task reassign tool. I have done this with some of my projects. I used one of the built in submit buttons, and re-purposed it to on click show a modal popup with a select user drop-down box populated from a database. When the modal was closed it wrote back the selected user to a hidden field that was then used by the workflow to re-assign the task. It has its limitations but it works. You can do it simpler by just having a drop-down list at the bottom, you don't need to go the full modal option for this to work.

Hopefully this makes sense i'm still not 100% awake yet haha, but if you need more information or an example let me know.

Good luck