Our Gemara tells us that there was a chamber in the Temple where invalidated stones from the altar were stored. These were stones that were removed from the altar as a result of having been contaminated through idolatrous worship by gentile kings during the time of the Hasmoneans.

Tosafos Yom Tov (Middos 1:1) questions how this is a proper rationale for demolishing part of the altar. The actual halakha is that something attached to the ground cannot be rendered invalid or forbidden, even when worshipped. Therefore, how could they allow for a dismantling of part of the altar when it was not necessary. After all, it is a prohibition to destroy parts of the temple. 

In an attempt to answer this question, he quotes Rambam in Laws of the Temple (1:17) that states it is only forbidden to dismantle or demolish parts of the altar when done with the intent of destruction. When done with the intention to repair, it is permitted (see Kesef Mishne ibid). Tosafos Yom Tov suggests that removing something that has an idolatrous tinge, even though it is not really forbidden, is still considered a form of repair. This is a stretch, because by definition, removing something which is not considered forbidden, it is not really a repair, as what was “fixed”? It would seem almost as if the Tosafos Yom Tov is using circular reasoning.

There is a Beis Yosef (YD 276:14) that also references this action of removing the stones and putting them in storage because they were contaminated. In my limited ability to understand, it would seem to me from the language of the Beis Yosef that he held differently than the Tosafos Yom Tov. He held that though the stones were not rendered forbidden due to the idolatry, they still were in a particular state called “pagum” which might be translated as “repulsive or metaphysically defective“. While this is a nuance, it still allows one to more easily understand how this was considered a repair and therefore permitted as opposed to a demolition. You might be able to say this was the intention of the Tosafos Yom Tov after all, though a careful reading into his words is necessary.

This is an important concept to become cognizant of, that even in deoraysa ideas there is a category between kosher and posul. There can exist items that are not posul but still not preferred for use, that is pagum.  In our everyday actions, what do we do that is kosher, but still pagum?