The reason he died physically was because he was ejected from the Garden and no longer had access to the fruit of the Tree of Life (Gen 3:22). The only purpose for the Tree of Life in the Garden was to keep Adam and Eve from dying so long as they were spiritually alive. That means that as created, they were physically mortal, that is, as created physical death was an integral feature of creation. They didn't die physically directly due to sin, but rather they died physically indirectly because when they sinned, they died spiritually in that instance and God would no longer grant them to eat of the fruit of the Tree of Life.
We, being descendants of Adam, have inherited his physical mortality which is, as I noted, an integral feature of creation. We have not inherited his "spiritual mortality". That is not a feature of creation but comes as a result of our own individual sin. In fact, the spirit of each individual human being is formed by God, Himself (Zech 12:1; Eccl 12:7) and is not in any way connected to Adam.