The Temple was the place where God communicated with his people, and where they sacrificed their offerings to him. They saw the Temple as God's revelation of Himself on earth; it was where God dwelt with them.
So what a shock it must have been when Jesus said, “As for these things that you see, the days will come when not one stone will be left upon another."