Remember, they had a lot of big leads which forces the other team to pass a lot. They would give up a lot of the short stuff and stop the big play (prevent D). When you just look at passing yards allowed that can be deceiving.
I'm not saying their Defense is Great, but I don't think it is as bad as people think. It has steadily improved over the season.