Our Gemara on Amud aleph records a declaration about the second Temple, as compared to the Temple, based on a verse in Chaggai (2:9) :

גָּד֣וֹל יִֽהְיֶ֡ה כְּבוֹד֩ הַבַּ֨יִת הַזֶּ֤ה הָאַֽחֲרוֹן֙ מִן־הָ֣רִאשׁ֔וֹן אָמַ֖ר ה׳

The glory of this latter House shall be greater than that of the former one, said GOD of Hosts

The Chasam Sofer (Derashos page 328) notes that since the second Bais Hamikdash did not have the dwelling of the Shekhina, the physical construct needed to be bigger. However, the first Temple was closer to a spiritual manifestation and therefore needed less physical presence.

This a consistent spiritual archetype, that the more spiritual something is, often it is less physically imposing. This is due to the inverse relationship between physical and spiritual; they do not easily co-exist. The Gemara Pesachim (50a) describes how the spiritual world and the physical world are often exactly the opposite: What is big here is small up there, and what is big up there is small down here. There is a saying from the Gemara (Bava Metzia 85b), “One coin in a pitcher cries out 'rattle, rattle." (Bava Metzia 85b) meaning to say, by comparison, a container filled with coins at the top will not make any rattling noise. So a person with a full brain makes less noise than an empty-brained person. There is a similar saying in Yiddish, “Fun leydike feser iz der lyarem greser. So too, sometimes the people that have the least to offer assert themselves the most.

It’s a fascinating and pervasive archetype. Mount Sinai the smallest of mountains, was chosen to be the place where Heaven meets Earth, and the Torah was given (Megillah 29a). There too, in reaction to the noisy clamoring of all the other mountains who vied to be selected for the honor of the Torah, the Gemara (ibid) says, “The one who makes the arrogant claims is actually the one with the blemishes.” Additionally, Moshe who should require a great oratory to give over the nuanced and inspiring teachings of the divine Torah, not coincidentally suffered from a speech impediment. Or all the matriarchs, whose critical divine destiny was to bear children who would be the progenitors of the Jewish people, suffered from infertility. That cannot be a coincidence. Rather, it is this idea that the spiritual can only come when the physical recedes and makes room for it. A person who could not naturally speak is ironically the only proper conduit for divine speech,because his physical efforts of speech would not interfere. Similarly, only a woman who could not have children naturally could be the proper receptacle for offspring that were divinely determined.