The story begins with Elias Hasket Derby, one of America's first self-made millionaires from the sea trade. When he died in 1799, his daughter, Elizabeth and her husband, Captain Nathaniel West, inherited his farmland in Danvers, Massachusetts. They built a country house on the property, sparing little expense.

In 1806, however, shortly after the house was completed, a bitter divorce forced Nathaniel off the property. After Elizabeth died in 1814, the house was left to her three daughters. When one of the daughters died in 1819, Nathaniel inherited one-third of the estate. He moved four rooms to Salem, establishing the core of what is 34 Chestnut Street today. To these four rooms, he added a hallway, a third floor and a back ell section.

In 1911, Anna (Wheatland) Phillips bought the house, and after fourteen months of extensive renovations, she, her husband Stephen Willard Phillips, and their five-year-old son Stephen moved in bringing with them five generations of family furnishings. Stephen W. Phillips inherited the house in 1938, upon Anna Phillips death. Upon Stephen's death in 1955, the house and its contents were left to their son. When he passed away in 1971, 34 Chestnut Street, which he described as a "veritable treasure house of antiques," became a historic house museum.