There is no flaw, the point of the challenge is to prove actual honor. It shows that I do willing to risk to prove it. If he refuses - it means he isn’t willing to risk it and thus doesn’t have one.
If I will call someone dishonorable, I will always be ready to face them and take responsibility for my words, if they will consider themselves honorable and will challenge me for that. If they won’t challenge me for that, it means they’re really dishonorable and I was simply right in my estimation.
There are three main components to honor: Loyalty, Courage and Honesty. If he doesn’t have Courage to answer for his deeds and words, then he surely can’t be considered as honorable.