The text, inked late Monday, was hailed as a breakthrough in a long-stalled national dialogue between political parties and the government aimed at drafting a new constitution for Yemen and preparing for elections in February, state news agency Saba reported.
The southern question has been a major stumbling-block for the talks launched in March, with hardline factions of the secessionist Southern Movement boycotting the discussions.
The dialogue is part of a transition backed by the United Nations and the Persian Gulf countries which saw President Ali Abdullah Saleh step down after 33 years in power following massive Arab Spring-inspired protests in the region's poorest country.
President Abdrabuh Mansur Hadi, who took over after Saleh agreed to step down in 2012, attended the signing ceremony alongside UN envoy to Yemen Jamal Benomar and delegates from key parties.
But hardliners of the Southern Movement boycotted the session.
The document was signed by representatives of 17 political parties and civil society groups who pledged to secure "a just solution to the southern question... on the basis of federalism," according to a copy seen by AFP.
But three political groups rejected the document on Tuesday, showing the divisions that linger around the issue.
Saleh's General People's Congress signed the document, but a faction within the party rejected it, as did the Yemeni Socialist Party and the nationalist Nasserist Party.
A committee formed by Hadi will now choose between forming two regions in the south and four in the north, or two large entities -- one northern and another southern.
The talks, which had been due to close on September 18, accepted the principle of a federal state, but stumbled over the number of regions that should be set up.
Hadi and northern delegates had suggested a federal state should comprise several entities while southerners have demanded a federal state made up of a north and south only.
Following the end of British colonial rule in 1967, southern Yemen was independent until union with the north in 1990.
Four years later, a secession attempt sparked a brief but bloody civil war, with northern forces retaking the south. Since then, southerners have complained of being marginalized by the government in Sanaa.
The document inked on Monday recognizes that unification caused "injustice" in the south and offers a mechanism to compensate southerners whose property was confiscated.
Benomar hailed the agreement as a "quantum leap in resolving the southern question," in remarks to AFP.
"This is a historic moment" and the new federal state which will be created in Yemen will mark a "complete break with a history of conflicts, oppression, power abuse, and mismanagement of resources," he said.
A source from the presidency in Sanaa told AFP that the committee will on Thursday present its final decision on the number of regions making up the future federal state.
"With this agreement, we approach the end of the national dialogue," said Benomar.
However, the situation remained tense in the south, where five people were killed on Tuesday when government forces dispersed a pro-secession protest in Ataq, capital of Shabwa province.
Three soldiers and two armed protesters were killed in the incident, Southern Movement members and a police source said.
Violence has intensified in south Yemen amid tribal anger over the killing of a local chief and his bodyguards at a checkpoint earlier this month.
BA/BA