That was more or less what we had before the removal of bounties and the “fix” to the previously use version that everyone exploited. The “it may not result in a nice payout for people who get the kill” was exactly the downside that this new system introduced and thus killed bounties even more than the exploitation before.
Any bounty system that allows people to kill other players who have not done anything wrong to deserve being killed just like that is only going to be abused.
Likewise, bounties for people who cannot be legally engaged without making it possible to kill them without ganking is also pointless because ganking potentially requires more ISK than people have bounties.
Limiting bounties only to players who have done something game-mechanically-wise that deserves their death is pointless because you can already engage them freely and they don’t need another flag to achieve that.
Unless you want a pointless system or an unreasonable free-for-all that kills other professions in the process, there is no solution to bounty hunting in a formal, game-mechanics way. The best thing you can do is issue bounties in this way: ★ ☠️ Headhunters' public killright office ☠️