According to the 1841 census, he was born about 1801, though I haven't been able to find any record of his birth. In March 1820 he married Mary Webber in Bures St Mary, Suffolk, a few months after the baptism of their first child, James.
The births of Louisa and Jeremiah followed and were both baptised on the same day in February 1823. Another child, Joseph Whybrew, was born in Bures St Mary in 1825 and died in March that year. It's not clear whether he was the child of James and Mary or another family. (There were other Whybrews living in Bures at the time.)
James was left a widower when Mary (Webber) Whybrew died at the age of 26 in April 1825. He remarried a year later in April 1826, again in Bures St Mary, to Sarah Baldwin. Their first child, Sophia, was born in 1826. It seems there were two Sophia Whybrews born in the same area that year, and they both appear (in different places) in the 1841 census in Essex. It's possible to follow each of them through the later censuses. James and Sarah's daughter Sophia married a Charles Duncombe in 1846. The second Sophia seems to have remained single and ended her days in an asylum.
Also with James and Sarah in the 1841 census were Eliza, born about 1832, Harriet, born about 1833, and David, born 1838. The family were living in Wormingford, Essex at this time, opposite the Crown Inn. James' occupation is listed as 'sawyer'.
|The Crown Inn, Wormingford|
© Copyright Robert Edwards under a Creative commons licence
Perhaps the family struggled to survive on a labourer's wages. A James Whybrew born about 1800 appears in the Essex courts in 1845 charged with larceny, for which he received 14 days imprisonment. But was this David's father? It could be any one of the three James Whybrews born in Essex between 1796 and 1806 who appear in the 1841 census.
In 1848 a James Whybrew died in the Lexden registration district (which includes Wormingford) but no age is recorded so again it's difficult to be sure that it's the same person. One clue that both James and Sarah had died before 1851 is that David Whybrew, still in his teens, was a resident of the Lexden and Winstree Union Workhouse at the time of the census. In that case, James lived less than 50 years.