It sees to me that unless some counter-thrusts are applied which yield a reactive force, like a booster or maneuvering thruster, the object would follow the geodesic w/o much impact.
We agree on that point fundamentally. There would be little felt impact on a swing trajectory using using gravity assist.