Palmer Park for many years was kept as a “white” space. Black residents were not permitted, and antagonism could be seen in several areas. The naming of the park after Benjamin Palmer—a staunch defender of slavery and leading segregationist—set an early tone. The ordinance changing the name from Hamilton Square, written in 1902 by the city council, praised Palmer and noted the name change was a “testimonial of esteem to the memory” of Palmer. Edgar A. Perilloux, historian for the Carrollton Men’s Business Association and author of the Carrollton Centennial booklet, defended the “white” history of the park. In reference to a popular idea about the park's origins, he wrote:
“Despite the fact that the official records prove the contrary, many people of the Carrollton section, including many who should know better, cling tenuously to the story handed down from one another, that Palmer Park was donated by some old negro.” His research demonstrated the property originally belonged to a white landowner.
Events in the park further demonstrated racial prejudice throughout the Jim Crow era. During a 1924 speech in Palmer Park, Shreveport Mayor Lee E. Thomas, challenging Senator Randsell for his seat, drew loud applause when he accused the senator of signing a letter supporting a black man for a federal job; the mayor's allegation sought to condemn the senator's egalitarian gesture. Similar racism could be seen in reaction to a 1934 incident. Residents nearby the park and civic organizations complained about an unlicensed shoe shine stand, "Sam's Shine Parlor," which appeared in the park. The stand, aimed at people waiting nearby for the bus to Kenner, was originally chained to a tree in the park. The black vendor's chair was removed. White vendors, like the man who sold hot tamales, were allowed in the park.
African Americans were denied access to “white parks” in New Orleans until the late '50s and early '60s. The Supreme Court had struck down segregation in public parks in 1955, but it was not until 1958 and the threat of federal intervention that City Park’s facilities were desegregated. The New Orleans Recreation Department (NORD), despite being recognized in 1949 by Life Magazine as “the most progressive in the U.S.,” did not desegregate its parks until it was ordered to do so by a judicial ruling in 1963. The judicial panel in that case, Barthe v. City of New Orleans, noted that NORD had built separate spaces for blacks after it was created in 1946; however, only 19 such spaces were provided for African Americans and over 100 for Whites. The review also commented that the parks were “by no means equal.” They argued the segregation was unconstitutional and ordered an immediate desegregation of all city parks. The city appealed, but The Supreme Court affirmed the ruling in a 6-3 decision in 1964. The desegregation of parks was one of the factors contributing to “white flight” from the city to the suburbs, and NORD lost substantial support and funding as a result.
In 1972, a proposal was put forward to incorporate Palmer Park into NORD, which would in effect integrate the park. Neighborhood residents opposed the measure and placed an advertisement in the Times-Picayune. The creators of the ad, identified as “Concerned Citizens of Carrollton,” argued the park would become a “public disgrace,” similar to Samuel Square, another park that had been integrated after a takeover by NORD. They warned that property values would plummet, adding, “No wonder 90,000 former residents of Orleans Parish have migrated to neighboring parishes,” referencing the white flight from New Orleans. The takeover by NORD was defeated in 1972, although the park did officially become part of the city system in 1977. Similar tones could be seen in letters to the Times-Picayune after integration complaining of children from outside of the neighborhood playing in the park.