The UN High Commissioner for Human Rights Navi Pillay said on Tuesday that the grave was uncovered in Bentiu, the capital of the oil-rich Unity State.
She said 34 bodies were found at the site, while there are "reportedly at least two other mass graves" in the capital, Juba.
The discovery follows more than a week of deadly fighting between government troops and rebels backing former vice president, Riek Machar.
On Thursday, Human Rights Watch said South Sudanese soldiers and rebels were carrying on ethnic killings.
The US-based group detailed cases where soldiers from President Salva Kiir’s Dinka tribe and rebels from Machar’s Nuer tribe had killed people only based on their ethnicity.
Official figures say the weeklong clashes have left some 500 people dead nationwide, but the real death toll is feared to be far higher, with witnesses recounting a wave of atrocities, including an orchestrated campaign of mass killings and rape.
Meanwhile, UN Secretary-General Ban Ki-moon asked the Security Council to almost double the size of the UN mission in the country and warned warring factions that the international body would look into reports of human rights violations.
“Those responsible at the senior level will be held personally accountable and face the consequences -- even if they claim they had no knowledge of the attacks," he said.
According to the UN, fighting has spread to half of South Sudan’s 10 states, with hundreds of thousands fleeing to the countryside for the fear of their lives.
One of the world's youngest nations, South Sudan gained independence in July 2011 after its people overwhelmingly voted in a referendum for a split from the North.
SHI/SHI