I'm sorry 3 seconds but you do not know the rule, yes the defense can down the punt but if you notice everytime it is done they pick up the ball or hold it until the ball is whistled dead.
If the ball is touched by a kicking team player with no possesion it is still a free ball for the receiving team and no matter what happens they can always choose to take the ball back at the illegal touch or first touching spot.
I am afraid it is you who does not know the rule.
Illegal touching is not really a penalty. There is no yardage given for it. All it does is ensure the recieving cant lose possession of the ball since the kicking team touched it first.
They dont blow the play dead, because the receiving team has the option to try & advance the ball, but even if they fumble they get it back at the spot of the original touching, 'illegal touch' by the kicking team.
Its just wording to say the ball was downed.
Its not really 'illegal', cause it is not a penalty.
Any legit or real penalty during play during the play by either team is enforced.
Now if the penalty is on the receiving team and is accepted, it is as if the illegal touch never happened & the foul is enforced from the previous line of scrimmage.
That is the RULE. The refs got it right.