He was married to Dorothy and they had five children. He was the brother of John, Lord Byron; Richard, Lord Byron; and Sir Thomas Byron.