It's a natural brain response. The brain hears sounds that it then transforms to get the message. So when it misses a bit, it "rewinds", trying to recall what has just been said, then catches on.
Now, most language students do something else. They add 2 phases. Instead of just hearing and understanding, they hear, transform into words, translate, and then understand. Some of them even visualize a written sentence before translating it.
This requires a huge effort. And the brain cannot go through this process AND hear and process the rest of the sentence.
So what can you do about it?
First, try to link the French words to images, feelings... not to the English words. Most of the time, you do not need to translate - it certainly works for basic concepts. For example, "Un chien": see the dog in your head instead of thinking "a dog". "J'ai faim": bring up the sensation of hunger instead of linking it to "I am hungry"... and you'll avoid the mistake of saying "je suis faim"...
Then, stay focussed on the context. Not only words can convey a message: looks, intonation, the rest of the sentence... When someone asked "tu veux du thé?" (would you like some tea?) if you're seeing what happens, you don't need to hear the question...
If you don't understand something, well, let it go... focus on staying in touch with the flow of the conversation. Chances are that the rest of the sentence will give you enough info to "fill in the blank". And if it doesn't, then it will be time to say "je n'ai pas bien compris" (I didn't get it all).
You'll find more advise on how to better your listening skills on this post.
And Also why it's important to learn French with audio and not only books.
And I'll help you pick the very best French learning method and tell you how to avoid scams
