The men were shot dead in the restored ruin, which had been used for an annual festival in the city, in front of the crowd on Wednesday.
The murdered locals were accused of being government supporters, according to a report from the Syrian Observatory for Human Rights.
'ISIS executed 20 men by firing on them in front of a crowd gathered in Palmyra's Roman theatre, after accusing them of fighting for the Syrian Army,' Rami Abdel Rahman, head of the Syrian Observatory for Human Rights, told AFP.
He added: 'ISIS gathered a lot of people there on purpose, to show their force on the ground.'
Supporters for the group also took to Twitter to boast that a number of people had been killed by the group inside the amphitheater, which forms part of the city's 2,000-year-old ruins which are a UNESCO World Heritage site.
The extremists took control of the city last week.
Already, they have killed at least 200 people and taken around 600 captive, according to the Observatory