Our Gemara on Amud Aleph tells us that the Esrog tree is so sensitive, that following the Shemitta year, and having all the people traveling through the orchard and touching the tree to Pick its fruit, it will not produce fruit again until the third year after Shemitta.

Mei HaShiloach ( מי שילוח, ליקוטי ש״ס, ראש השנה טו ) comments on this that the Esrog which is destined to be held in the human hand for a mitzvah is extra sensitive to profane touching.

Sod Yesharim (Succos 40) goes further with this idea. He says the Esrog is a durable hardy fruit that stays on the tree the entire year, as the Gemara says. And yet, it also is weak and sensitive to the extent that it takes years to recover from the foot (and hand!) traffic of Shemitta. He says this paradox of the Esrog is comparable to humans versus animals. Humans are the most vulnerable to the forces of nature, far more than animals who can live out in the wild, and also the most capable of dominating nature. The human intellect and psyche primes its vulnerabilities as well as it’s strengths and they are inseparable.

Spiritual matters can be violated and destroyed by the coarse human hand. This is true for matters that are holy objects such as tashmishey kedusha like sifre Torah, Tefilin and Mezuzos as well as human sacred areas such as the sexual parts of our bodies and souls. If they are violated and they lose their sacred quality, it can take time for it to grow back.