De La Warr became the first hereditary peer to join the Labour Party, and in February 1924, then aged 23, was one of the youngest ever ministers when he was appointed Lord in Waiting in the first Labour government of Ramsay MacDonald. He made his maiden speech in the House of Lords the same month. In the second Labour government of 1929 to 1931 he served as Captain of the Honourable Corps of Gentlemen-at-Arms and Under-Secretary of State for War between 1929 and 1930, as Parliamentary Secretary to the Ministry of Agriculture and Fisheries between 1930 and 1931 and as a Lord-in-waiting between 1929 and 1931. In 1931 the Labour government fell and MacDonald formed a "National Government" of politicians drawn from all parties. De La Warr was one of only a tiny handful of Labour ministers to follow MacDonald, and prior to the 1931 general election, he was instrumental in the formation of the National Labour Organisation to provide a vehicle of support for MacDonald and other ex-Labour members of the National Government. He resumed office as Parliamentary Secretary to the Ministry of Agriculture and Fisheries, a post he held until 1935, and then served under Stanley Baldwin as Parliamentary Secretary to the Board of Education between 1935 and 1936 and as Under-Secretary of State for the Colonies between 1936 and 1937. In 1936 he was sworn of the Privy Council. In 1937 the new Prime Minister Neville Chamberlain gave De La Warr his first cabinet post as Lord Privy Seal. Like several other younger members of the cabinet, De La Warr found himself disagreeing over the government's foreign policy, and contemplated resigning over the Munich Agreement but decided not to do so. In the aftermath of the agreement he was transferred in 1938 to be President of the Board of Education. During his time in this post it was expected that he would oversee legislation for raising of the school leaving age to 15, but the outbreak of World War II deferred all such plans until the end of hostilities. In April 1940 De La Warr became First Commissioner of Works in a series of ministerial changes by Chamberlain, but was demoted from the cabinet. The following month Chamberlain was replaced by Winston Churchill, who formed an all-party coalition government, and the objections of the Labour Party to National Labour ministers meant that De La Warr was dropped, and he did not return to government for eleven years. In 1951, in Churchill's peacetime Conservative government, De La Warr was appointed Postmaster General, and as such, was in charge at the time of the Eastcastle Street robbery, before leaving office for the final time in 1955. He continued to contribute regularly in the House of Lords until 1966, but from then on until his death ten years later only spoke twice, both times in 1972. Apart from his career in national politics, De La Warr was Mayor of Bexhill-on-Sea between 1932 and 1934 and a Justice of the Peace and Deputy Lieutenant for Sussex. In 1956 he was made a Knight Grand Cross of the Order of the British Empire. The De La Warr Pavilion was built in 1935 in Bexhill-on-Sea and was named after Lord De La Warr. The "De La Warr" in both the pavilion's name and the Earl's name is pronounced "Delaware".
Lord De La Warr died in January 1976, aged 75, and was succeeded in the earldom by his elder and only surviving son, William. The Countess De La Warr died in June 1992. One of his sisters was Lady Avice Ela Muriel Sackville. He attended her marriage to Stewart Menzies,, dressed in a lowerdeck seaman's bell-bottomed uniform.