In 1847, the State of Maryland gave permission to the United States War Department to construct a fort in the shallow water of Soller's Point Flats to protect the city of Baltimore. Fort Carroll was important for the defense of Baltimore—before the fort's construction, Fort McHenry just outside the city was the only military defensive structure between Baltimore and the Chesapeake Bay. The fort was part of the "Permanent System" or Third System construction program, which aimed to defend America's most important ports. Then Brevet-Colonel Robert E. Lee designed the hexagonal structure and supervised the construction, which the U.S. Army Corps of Engineers commenced in 1848. The fort received its name on 8 October 1850. In 1852, Lee left Baltimore to become Superintendent of the United States Military Academy at West Point. In 1853 a lighthouse, now abandoned, was built on the ramparts to aid navigation into Baltimore Harbor. In 1898 a new lighthouse was built, the one still seen today. It was automated in 1920 and discontinued operations before 1945. The original design foresaw the fort being armed with some 225 cannon on three levels. However, in April 1861 at the outbreak of the Civil War, Fort Carroll's walls were still less than half the planned height of thirty feet. Only five gun platforms were ready and only two were armed with guns. Still, the Army emplaced about thirty cannon and manned the fort throughout the war. In April 1864 torrential rains flooded the fort's magazines, which led the Army to move all the powder and ammunition to Fort McHenry.
When the United States entered into the Spanish–American War in 1898, the Army again manned the fort's batteries, which were by then completely obsolete. The Army therefore commenced the construction of modern concrete gun emplacements following the designs of the Endicott Board. The Army created three batteries: Battery Towson, Battery Heart and Battery Augustin. The new batteries were ready by September 1900, well after the end of the war. They received their names on 30 March 1903.
After World War I broke out, in 1917 the Army removed the guns from Battery Heart and in 1918 those from Battery Towson to use elsewhere. By 1920, all guns had been removed from the fort. In March 1921 the Army officially abandoned Fort Carroll and moved whatever military equipment was left to nearby Fort Howard. The War Department declared the island excess property in 1923, but took no immediate steps to sell the land. A variety of proposals for the use of the island were advanced, including a prison, as well as a 1923 plan advanced by Baltimore mayor William Broening to place an electric "Welcome to Baltimore" sign on the island, accompanied by a statue of Lord Baltimore. In World War II the Army used the fort as a firing range. It also served as a checkpoint for vessels. In May 1958, Baltimore attorney Benjamin Eisenberg purchased the island for, intending to put a casino there, but development plans never materialized. The fort is now deserted and the habitat for various animals. It is also a site for occasional urban explorations. In 2013, Preservation Maryland placed Fort Carroll on its list of threatened historic properties. It was listed on the National Register of Historic Places in 2015.