It depends on what version of Advent Children you're using. The DVD is standardized to 29.973 fps, but unfortunately I've found that the footage is interlaced. The most likely reason for Vegas to cause more of a 'blend' effect from the image to the mask is because it's trying to resample it, (without even being set to resample). I've had this happen before with a ton of anime and, believe me, there's no quick fix. >< If you've already done what's been said above, (disabling resample, setting to progressive, etc.,) and you're still having issues, my only suggestion without completely reprocessing the footage into something progressive is to mask it, split each frame, and then adjust the mask to the frame as it appears at the very beginning of the frame, (and be sure to delete the keyframe at the very end of said frame).
Blu-ray, on the other hand, is set at 60 fps. If you're having issues with it, try setting it to 30. If it still happens, then they REALLY went cheap on converting it and I would do the same thing as above.
Hope that helps!