The library is a one-story three-by-three-bay structure faced in locally quarried uncoursedpuddingstone and topped with a Mission-style ceramic-tiled hipped roof that overhangs the building on all sides and covers the front steps as well. A stone chimney, similar to the wall, rises from the west side. A wheelchair ramp leads into a rear entrance; a short set of stairs to the front. Inside, there is one single room with a large stone chimney breast and fireplace and mahogany bookshelves. Other original wooden trim and furnishing remains. There is a bronze memorial plaque with a poem by Jane Rushmore Patterson, Rushmore's daughter. The basement is fully excavated.
History
Woodbury's public library began in the years following World War I as a small collection of books kept first on a shelf at the local drugstore and later moved to a grocery store and the telephone company office as it got larger. Jane Patterson encouraged her parents to help build a permanent home for the library, and they bought the land and put up the money to build it. The deed stipulated that the building and property remain in use as a library or some related function, or it would revert to the Rushmore family. Howard Gregory, a young local war veteran and architect whose father's firm had built the Rushmore home in 1906, designed the library and oversaw its construction throughout 1924. One of his early renderings is displayed in the building today. His use of uncut stone was in keeping with both the principles of the American Arts and Crafts movement and other nearby rustic stone architecture, such as the gatehouse at the F.F. Proctor estate a few miles away and the structures built for Harriman and Bear Mountainstate parks and the new gated community of Tuxedo Park. Charles Rushmore died in 1931, and eight years later the library would be formally renamed in his memory. At the same time it was deeded to what was then the Highland Mills Common School District. In 1951, that district was one of several merged into the new Monroe-Woodbury Central School District. The library remained under the new district's control for five years, until in 1956 the school district transferred it back to the town. It became Rushmore Memorial Public Library, the Highland Mills branch of the town library, in 1958. It was a charter member of the regional Ramapo-Catskill Library System when that was established the next year. In 1966, all volunteer services at the library were ended. The wheelchair ramp was added in 1970, with a window converted to a door in the process. Fifteen years later, in 1985, the town built a newer, larger library facility, and it briefly considered whether to keep the original building or not. It was decided that the historical society would keep collections related to local history there. At that time, the historical society had tried twice to get the library listed on the National Register, but both applications failed. In 2008, with help from New York'sOffice of Parks, Recreation and Historic Preservation, they were successful.
