I'm going to put in fresh fasteners every time I pull a door panel, That's all I really can think of thus far... I'll also be deadening the panels themselves when I strip the whole car out this summer... should be... lots of fun. You could also probably add some more padding behind the panel to make them a bit more solid to begin with.
And as far as hearing damage... as far as I've seen thusfar, a car is much less durable than my ears.
Last hearing test I took I still had perfect hearing, but my sedan is falling apart.