He was born to a Jewish family in New Brunswick, New Jersey, and grew up in Plattsburgh, upstate New York, where his parents owned a chain of women's wear stores. When he was 16, his family moved to Forest Hills, Queens, and he later attended Columbia University and Columbia Law School. In 1960, he started work as an unpaid intern for a law firm working on the estates of Charlie Parker and Billie Holiday. Over the next few years he developed a love of jazz music, did some legal work for Moe Asch at Folkways Records, and began advising jazz and rhythm and blues musicians on copyrights and contracts. He also learned Esperanto, and made a recording of poetry and songs, Ni Kantu en Esperanto, to promote the language, releasing it on his own label which he called ESP-Disk. After receiving funding from his parents, he decided to establish the record label as a serious venture, to promote non-commercial and experimental music which would not otherwise be heard. In 1963, he approached free jazz pioneer Albert Ayler in a Harlem club, and offered to record an album with him. The album, Spiritual Unity, became ESP's second release. Thereafter, he recorded such jazz musicians as Sun Ra, Pharoah Sanders, Ornette Coleman, Paul Bley and Gato Barbieri. He also recorded writers including William Burroughs and Timothy Leary, and underground folk and rock acts including The Fugs, The Godz, Pearls Before Swine and The Holy Modal Rounders. Stollman has faced allegations of not paying royalties to artists, signed to ESP-Disk. Tom Rapp of the band Pearls Before Swine claimed that "We never got any money from ESP. Never, not even like a hundred dollars or something. My real sense is that he was abducted by aliens, and when he was probed it erased his memory of where all the money was". Peter Stampfel of the band Holy Modal Rounders and The Fugs claimed that Stollman told him that "the contract says that all rights belong to me. You have no royalties ever, ever, ever. The publishing is mine. You don’t own the songs anymore. We don’t owe you anything". Members of The Fugs have also stated claims that they received an unfavourable record contract. Ed Sanders said that "our royalty rate was less than 3%, one of the lower percentages in the history of western civilization". 801 Magazine, which featured an interview with Stollman in 2008, said that Stollman claimed that "he paid all the recording costs and gave the musicians small advances", and that "he never made any money from the music". For an extended period Stollman pretended to represent the Sun Ra estate. Irwin Chusid, who has served since 2013 as administrator of Sun Ra LLC, said in a 2017 interview: "I found the man crazy, creepy, and vulturous. He tried for years to get the Ra heirs to allow him to represent the estate, but they rejected him each time. So in 2001 he drafted... bogus document managed to convince a number of foreign sub-publishers, record labels, and performing rights organizations that he was the legit administrator. He pocketed all foreign and some domestic revenue for 13 years." On the positive side, Stollman had artists sign a two-page contract for only one record, so that they could consider offers from larger labels if the album proved successful; and he gave them joint ownership with ESP, allowing for an equal partnership in contrast to the standard arrangement where a label has sole ownership of an album. It set, he claimed, “a new standard for the treatment of artists.” ESP album sleeves contained the message "The artists alone decide what you hear on their ESP Disk". Although many of his label's releases were critically acclaimed, most did not sell well, and by the early 1970s his funds had been exhausted. Stollman married, moved to live on a farm in the Catskill Mountains, worked as a lawyer, and closed ESP-Disk in 1974. In the 1980s he worked as an assistant New York Attorney General, retiring in 1991. In 2005, he re-activated the ESP label, to reissue old recordings as well as making new recordings. He died of complications from prostate cancer in 2015.