Rhinelander was born in 1869. He was the youngest of eight children born to Frances Davenport Rhinelander and Frederic W. Rhinelander, the president of Metropolitan Museum of Art. Frederic William Rhinelander, His older brother, Frederic W. Rhinelander Jr., was married to Constance Satterlee, a daughter of Bishop Henry Y. Satterlee. Through his sister Ethel, he was the uncle of Frederic Rhinelander King, a prominent architect with the firm of Wyeth and King. His paternal grandparents were Frederic William Rhinelander and Mary Lucretia "Lucy Ann" Rhinelander. His maternal grandparents were the Rev. Thomas Harvey Skinner and Frances Louisa Skinner. Through his maternal aunt, Lucretia Stevens Rhinelander, the wife of George Frederic Jones, he was a first cousin of novelist and decorator Edith Wharton and Frederic Rhinelander Jones. Through aunt Mary Elizabeth Rhinelander, the wife of Thomas Haines Newbold, he was a first cousin of New York State SenatorThomas Newbold. Rhinelander was educated at St. Paul's School in Concord, New Hampshire before graduating from Harvard University with an A.B. degree in 1891, from Oxford with another A.B. degree in 1896, and from Oxford again with an M.A. degree in 1900.
Career
After he finished at Oxford, he was ordained a deacon at Calvary Church in 1896, in New York City, by his older brother's father-in-law, Bishop Satterlee, the first Bishop of Washington who had been his rector in New York. He followed Satterlee to Washington where he worked under him at St. Mark's Church, a parish and mission for seven years. In 1903, due to health issues, he ended parish work and became a professor of pastoral studies at the Berkeley Divinity School in Middletown, Connecticut. He was at the School for four years, during which time he was married, and then in 1907, became the new chair of the History of Religion and Missions at the Episcopal Theological School, then in Cambridge, Massachusetts. In 1909, he was offered the vicarship of Trinity Church in Manhattan, "one of the most important of the ten houses of worship then conducted by Trinity parish, New York" but declined. On May 10, 1911, he was elected Bishop Coadjutor of Pennsylvania, and later, succeeded the Rt. Rev. Alexander Mackay-Smith as the 7th Bishop of Pennsylvania. Rhinelander served in this role until May 1, 1923 when he resigned due to poor health. He was succeeded by Thomas J. Garland in an 1924 election. Along with Bishop James E. Freeman, he was instrumental in the creation of the Washington Cathedral, which he later served as a trustee of and where he was a Warden of the College of Preachers.