I have had this happen extensively in the Galactic conquest (campaign) mode. (Ver. Full 1.56, GC only add on)
It has a tendency to happen during a multiple battle turn. The chance goes up to 80% when one of my fleets after the first battle has captured ships.
After the game locks-up -> ctrl+alt+del -> close game -> start game -> reload save, the ships can be placed on top of each other. If the ships can be placed on each other the game will lock when fight is clicked no matter the composition of the fleet. Sometimes clearing the ships off the deployment board and putting them back on will fix it.
Auto deploy will sometimes place ships on the board so that they are colored red. Going with red ships will lock the game every time.
I currently have a savegame with no mods that has this problem.