In 1716, John Sargeant purchased 120 acres of land on Spring Hill from Thomas Huntington.  Sargeant was a leading citizen of Mansfield (Pond Place) and a signer of the church covenant.  Acreage was given by John to his son, Isaac, on the latter's marriage.  The house was probably built in 1734, as recorded on a beam in the attic.

In 1794, on the death of Isaac's widow, Ann, the house and 89 acres were purchased by Dan Storrs, who sold the land to Azariah Freeman.  The house remained in the Freeman family for the next 113 years (from 1795 to 1908).  The noted 19th century miniaturist, George Freeman, is believed to have lived in the house for a time.

In 1951 Edith McComb purchased the house and named it Altnaveigh, Gaelic for "hill top".  For much of the 20th century, this historic landmark has been utilized as an inn and restaurant.

Altnaveigh was purchased by its present owners, Douglas and Gail Parks, in 2004. An ambitious renovation is currently underway to restore the house to its original grandeur and to modernize the kitchen facilities.