Our Gemara Amud Aleph to Beis makes an assertion:

כׇּל טוֹבָתָן שֶׁל רְשָׁעִים רָעָה הִיא אֵצֶל צַדִּיקִים. שֶׁנֶּאֱמַר: ״הִשָּׁמֶר לְךָ פֶּן תְּדַבֵּר עִם יַעֲקֹב מִטּוֹב עַד רָע״, בִּשְׁלָמָא רַע — לְחַיֵּי, אֶלָּא טוֹב אַמַּאי לָא? אֶלָּא שְׁמַע מִינַּהּ: טוֹבָתָן שֶׁל רְשָׁעִים — רָעָה הִיא אֵצֶל צַדִּיקִים.

Every act that is a benefit for the wicked is a disadvantage for the righteous, as a righteous individual gains no pleasure from this so-called beneficial act. As it is stated by God to Laban: “Take heed to yourself that you speak not to Jacob either good or bad” (Genesis 31:24). Granted, speak no bad; this is rightly so, i.e., understandable. But speak no good? Why not? Rather, learn from here that even something that would be a good benefit to the wicked like Laban, is a disadvantage for the righteous.

Likutei Moharan 30:7 discusses this idea in an interesting manner:

He says that the great Tzaddikim do not just have difficulty with the wicked, but even us regular folk who make efforts at piety, are still experienced by them as painful.  He gives as an example in terms of our prayers.  When we enter shul to pray, though we have the best of intentions, our lack of spiritual sensitivity is disruptive to those on a higher spiritual level and they must suffer under the noxious aura we give off.  

Through he means it in a spiritual sense, that somehow we disrupt the environment, it also can be understood literally. Even if we do not talk in shul, it is possible that a relative lack of sensitivity to decorum is still something that affects others.  We might tap our fingers, browse through sefarim, move about to and fro, which is typical but I can see how that might disturb another person’s concentration and sense of the sacred.  However, it is the next part of the Likutei Moharan I want to focus on.  He suggests that even in our own efforts to be good, bad things come up and disrupt.  He refers to the frustrating phenomenon of all the strange and irrelevant thoughts that come to mind when we try to clear our heads and pray.

There is a famous story about a chassid who shared with his rebbe that he was worried over a certain Gentile whom he owed money from long ago.  He was afraid that he would remember the forgotten debt and come after him. The rebbe quipped, “Not a chance.  After all, Goyim don’t daven Shemoneh Esre!”

Likutei Moharan goes on to offer a metaphysical explanation for this kind of distraction, which I think can also be seen psychologically. He says that once we devote ourselves to spiritual pursuits, all the misused and contaminated aspects of our thoughts bubble to the surface in order to achieve a proper redemption and have their holy sparks released.  Sometimes it is so easy to see how mystical phenomena and psychological phenomena are two sides of a coin, with both trying to explain an inner psychic process. In psychological terms, when you clear your mind, all the unresolved issues and concerns start coming to the surface in search of a resolution.  

Regardless of the explanation for the phenomenon, developing focus and concentration for clear prayer is a process that takes years of practice.  It is why Shulkhan Arukh (OH 93) codifies that one is required to pause and meditate for a period prior to, and after Shemoneh Esre.