In "Biblical Perspective" I.C. Holland writes:

Jesus is that living stone! Though He was rejected by men, He was chosen and approved by the Father who said, “This is my beloved Son, in whom I am well-pleased” (Matt. 3:17). The word for stone here is different from the title Jesus gave to Peter. There it was petros, a pebble or field stone. Here it is lithos, meaning a “worked” or shaped stone. The words living and lively indicate activity, growth, and progress, and stone represents permanency. The infantile state discussed in verse 2 has developed from “born” and “grow” to “tasting,” or experiencing, the grace of God in verse 3. Now in verse 4 coming shows further action that leads to built up in verse 5. At this point the figure changes to a spiritual house and holy priesthood, and in Ephesians 2:22 to “a habitation of God through the Spirit.” What a quantum leap! We become not only ministers of Christ, but ministers to Christ! Spiritual work is done by spiritual people.

Source: Building Christian Relationships: Adult Teacher's Insights, page 18.