The surveillance planes appeared to be "fiery objects" high in the sky, it said.
"At this time, no one believed manned flight was possible above 60,000 feet, so no one expected to see an object so high in the sky," it said.
The commercial pilots and other observers on the ground wrote letters to an Air Force unit in Dayton, Ohio charged with investigating such sightings.
Anxious to avoid exposing the ultra-secret U-2 program, Air Force officers explained the sightings as merely due to natural phenomena, though they knew the high-flying U-2 was the true cause.
U-2 and other surveillance flights "accounted for more than one-half of all UFO reports during the late 1950s and most of the 1960s," it said.
The 400-page report, titled "Central Intelligence Agency and Overhead reconnaissance: The U-2 and Oxcart Programs, 1954-1974," was released as a result of a Freedom of Information request dating to 2005 from the National Security Archives at George Washington University.
The study was published in classified form for spy agencies in 1992 and a heavily censored version was published in 1998.
Area 51's location has been an open secret for years but government documents released previously had not acknowledged its existence and role in such a detailed way. Officials also had referred to a location "near Groom Lake."
The CIA report said at the time officials decided to nickname the site "Paradise Ranch" to make it sound more attractive to potential workers.
The in-house history refers to Area 51 in passing, as the report is devoted mainly to recounting how the CIA developed the U-2 "Dragon Lady" and other "eyes in the sky" to spy on the Soviets.
Other stealthy planes have been tested at the site, including the SR-71 Blackbird, the F-117A fighter and the B-2 bomber.