Since that's a corner case which is not normally even looked at in grammar sources (because technically, it's spoken English with incomplete phrases and so on) - I guess it'll fix itself if you just practice more. Similar things are probably pure language intuition. And at least it didn't affect the meaning.