About 1520 James joined the household of Cardinal Wolsey, who praised him as a young gentleman "both wise and discreet". In early 1522, it was proposed by King Henry VIII that he marry his cousin Anne Boleyn, who was the great-granddaughter of Thomas Butler, 7th Earl of Ormond. This was to resolve a dispute her father Thomas Boleyn had with James's father Piers over the Ormond inheritance and title; Wolsey himself supported the proposal. The marriage negotiation came to a halt for unknown reasons. He subsequently married Lady Joan Fitzgerald some time before 21 December 1532. Lady Joan was the daughter and heiress of the other great Munster landholder, the 10th Earl of Desmond and his wife Amy O'Brien. James and Joan had seven sons:
During the early 1540s he gradually restored the Butler dynasty to their former position of influence, leading to antagonism from the quarrelsome Lord Deputy of Ireland, Sir Anthony St Leger. St Leger gave Ormonde command of the Irish forces in the Anglo-Scottish War of 1544. On the face of it this was an honour, but allies of Ormond accused St Leger of deliberately sending Ormond into danger. Ormond himself demanded an inquiry into claims that St Leger had planned his murder, and the matter was thought to merit a Privy Council investigation; the Council found in favour of St Leger and he and Ormond were ordered to work together amicably for the future. Key allies of Ormond like John Alan and Walter Cowley were removed from office, and Ormond was struggling to maintain his standing when he was poisoned.
Poisoning
On 17 October 1546, James was in London with many of his household. They were invited to dine at Ely Palace in Holborn. He was poisoned along with his steward, James Whyte, and 16 of his household. He died nine days later, on 28 October, leaving Joan a widow in her thirties. It is surprising, in view of Ormond's high social standing, that no proper investigation into his death was carried out. Who was behind the poisoning remains a mystery. His host at the dinner, John Dudley, 1st Duke of Northumberland, though he could be notably ruthless towards his enemies, seems to have had no motive for the crime, as he is not known to have had any quarrel with Ormond. A recent historian remarks that it would be an extraordinary coincidence if St Leger had no part in the sudden and convenient removal of his main Irish opponent.
