The demonstration came on Saturday in the Parade Square as people in the south were marking the anniversary of the end of British colony in 1967, when an independent state was created that lasted until the union with the north in 1990.
The protesters waved flags of former South Yemen and carried banners with pro-independence slogans.
"Through this demonstration, which is open to all groups without exception, we want to advance towards independence," said Fuad Rashed of the Southern Movement, adding, "We are determined to pursue our peaceful struggle to achieve our demand."
The demonstration came as the Yemeni President Abd Rabbuh Mansur Hadi said in a speech on Friday that there could be no question of statehood for the south.
"Yemen's unity will continue to be a sacred value," he said, adding that the southerners who support independence lived in a world of "illusion."
North and south Yemen unified in 1990 after the southern government collapsed. But four years later, the south tried to break away and this led to a civil war. The conflict ended with northern troops taking control of the south after winning the war.
The Southern Movement gained strength during mass demonstrations which forced former Yemeni president, Ali Abdullah Saleh, to step down in February 2012.
The secessionist movement calls for autonomy or the complete independence of the south. Southerner residents complain that they have been economically and politically marginalized by the central government.