The first dwelling house at Hickory Hill was built and the garden begun in 1820, when William Fanning Wickham, son of John Wickham, a notable attorney in Richmond, and his wife, Anne Butler (née Carter) Wickham, who was born at Shirley Plantation, made it their home. Her sister, Anne Hill (née Carter) Lee, was the mother of Robert E. Lee.
Their son, Williams Carter Wickham (1820-1888), became a notable lawyer, judge, politician, and an important Confederate cavalry general who fought in the Virginia campaigns during the American Civil War. He served various political posts representing Hanover County before and after the American Civil War. After the war, In November 1865, he was named the President of the war-ravaged Virginia Central Railroad, which ran westerly from Richmond. Soon, the Virginia Central was merged with the Covington and Ohio Railroad to become the Chesapeake and Ohio Railway (C&O), with the goal of completing a railroad link to the Ohio River. Williams Wickham is credited with attracting transcontinental railroad builder Collis P. Huntington and fresh financing from New York City to complete the task by 1873. Wickham remained active with the C&O through a receivership and financial reorganization, and was at his office in Richmond working when he died in 1888.
On June 26, 1863, General Robert E. Lee's son Rooney Lee was captured by federal forces at Hickory Hill. Rooney Lee was the husband of Charlotte Wickham, a cousin of Williams Carter Wickham.
The original house at Hickory Hill was destroyed by a fire in 1875, and replaced. Hickory Hill produced wheat (its major crop), corn, oats, and other fruits and vegetables. Unlike other Hanover County plantations, which sold locally, Hickory Hill sold its produce in Richmond where it brought a higher price. It had its own stop, Wickham, on the former Virginia Central Railroad.
