There is no documentation for Joe's age. His inclusion as an adult in the 1786 Mount Vernon slave census implies that he was then over age 14. Joe's wife, Sall was born around 1769, and worked as a seamstress at Mount Vernon. Joe may have been about the same age. At the time he came to Philadelphia, they had three young sons Henry, 7, Elijah, 3, and Dennis, 1.
Sall and their children were owned by George Washington and freed after the president's death. They took the surname "Richardson." Joe was a "dower," and would have been one of the 153 enslaved Africans inherited by Martha Washington's grandchildren after her death in 1802. It is not known which.
Although he remained enslaved, Joe and Sall Richardson managed to stay together and had at least seven children, all of whom were free. Two of their sons were working at Mount Vernon in 1835.