Yes, the ordering matters for the other branches in an exclusive gateway. If you have a default branch, it is the "last resort" unless any of the previous conditions are met. You can still choose the order of how the other conditions are evaluated. Consider the following case:

Branch 1: Field > 10

Branch 2: Field > 0

Default Branch: Default

If you prioritize branch 1, then 2, if the field is 20, it would follow branch 1. If the priority was swapped, it would evaluate branch 2 first and it would follow branch 2. In the second case, branch 1 would never be followed because branch 2 would always evaluate to true first. 

But if your question is about the ordering of the default branch, then no, that doesn't matter. It will only be evaluated at the end.