The ride will become somewhat rougher in potholes and rough roads because the wheel tries to go through the bump instead of over it. You may feel this in your hands.
Caster setting way above production spec will also upset bump steer because the outer tie rod (on a front steer car) will go up as the caster increases.
Extreme caster is also pegged as a high-speed shimmy causer but I really think this more pertains to solid axles and shopping carts more than short arm-long arm independent susp cars.
Super high caster will also ship diagonal weight as the outside wheel picks up and the inside pushes down. Depending on the cars set up increasing outside rear wheel weight may or may not be a good thing.
All that being said I subscribe to the more is better camp.
When you align the car have them pull the nose up and down to watch the toe change. Depending on what you see, loosening the steering box and idler arm and pushing them up at their pivots might help.