It sounds as though you will have irregular enough mating area not to allow for even a small lip flange running around the shroud. Any number of construction fireblock foams are available, but the colors are garish (by design) and not easily contained to a trim bead. I've never used them, but a spray type RTV gasketmaker may work. Last idea: A long-legged duct transverse joint that's coped and RTV'ed to the irregular surface, then allowing a straight cut shroud slipped in. Added benefit is the shroud is removable.