Mackal is widely considered to be one of the seminal figures in the subculture of cryptozoology, a pseudoscience. According to writer Daniel Loxton and paleontologist Donald Prothero, "Cryptozoologists have often promoted 'Professor Roy Mackal, PhD.' as one of their leading figures and one of the few with a legitimate doctorate in biology. What is rarely mentioned, however, is that he had no training that would qualify him to undertake competent research on exotic animals. This raises the specter of 'credential mongering', by which an individual or organization faints a person's graduate degree as proof of expertise, even though his or her training is not specifically relevant to the field under consideration." Along with ecologist Richard Greenwell and Belgian zoologist Bernard Heuvelmans, he was one of the founding members of the International Society for Cryptozoology, which was created in 1982 at the National Museum of Natural History in Washington, D.C., with the hopes of bringing a degree of respectability to what is often seen as a pseudoscience. The organization published a quarterly newsletter and an annual journal, and members met annually at meetings held at universities throughout the world. Mackal was the ISC’s vice-president for the entirety of its existence, although the organization gradually folded in the early 21st century owing to lack of funding and the deaths of Heuvelmans and Greenwell. Mackal's 1980 book Searching for Hidden Animals was negatively reviewed in the BioScience journal as a "reflection of mankind's anxiety in the modern world of science and an obvious extension of current interest in the paranormal."
Mackal began investigation into the Loch Ness Monster phenomenon during the 1960s. In 1965, he went to the Scottish Highlands and met several members of the Loch Ness Investigation Bureau, who were monitoring the loch in observation vans in hopes of seeing the creature. Fascinated by their work, Mackal began monitoring the waters himself and, after raising money in America, became the scientific director for the project, a position he held until 1972, when the Bureau disbanded. During this time, the LNIB conducted sonar probes of the waters near Urquhart Bay and installed underwater strobe cameras with the hopes of providing evidence of the Loch Ness Monster. Mackal also designed a "biopsy harpoon", a dart-like contraption he attached to a submarine in order to collect tissue samples. The team never had an opportunity to use the biopsy harpoons, and were unable to provide any conclusive evidence that the Loch Ness Monster existed. However, Mackal himself was convinced that something lived beneath the waters after recording his own sighting of the creature in 1970, and in his 1976 book The Monsters of Loch Ness, he suggested that a population of large, previously unknown amphibians were living in the loch. Mackal later changed his mind and proposed that the creatures were zeuglodons, serpentine whales believed to have gone extinct several million years ago.
During the 1980s, Mackal turned his attention to purported creature, the Mokele-mbembe, which figures such as Young Earth creationists claim is a living dinosaur in the Likouala swamp region of the Republic of Congo. Accompanied by University of Arizona ecologist Richard Greenwell and Congolese biologist Marcellin Agnagna, Mackal undertook two expeditions, the first in 1980 and the second in 1981, to find and photograph the creature. Mackal himself did not actually see the creature, but he and his colleagues did collect multiple firsthand reports from Congo natives, who, according to Mackal, consistently described a creature similar to a long-necked sauropod. During his interviews with the natives, Mackal also heard anecdotes about the Emela-ntouka, another possible living dinosaur which supposedly resembles a Monoclonius or Centrosaurus, the Mbielu-Mbielu-Mbielu, which is said to resemble a Kentrosaurus, and the snake- or lizard-like Nguma-monene. In 1987, Mackal wrote a book about his adventures in the Likouala swamps called A Living Dinosaur? In Search of Mokele-Mbembe. He had tried to obtain funds for a third expedition to the region, but his plans were never realized. Mackal has said of his interest in the Mokele-mbembe, “I admit that my own views are tinged with some romanticism, but certainly not to the extent that I would endure extreme hardship, even risk my life, to pursue a dream with no basis in reality.”