Francois Crepeau, the UN Special Rapporteur on the human rights of migrants, made the remarks at a news conference in Doha on Sunday at the end of his weeklong visit meant to assess the human rights of migrants in the country.
He stated that living conditions of foreign workers in Qatar are poor and described one compound he had visited as a "slum".
"This marks a stain on Qatar's reputation and is something that can be improved right away," he said.
Crepeau also said the issue might get even worse as more migrants could come to Qatar for work on the construction sites for the 2022 World Cup.
Foreign workers cannot change jobs or leave the emirate without the permission of their sponsors, who are often Qatari companies or individuals who provide workers to businesses for profit.
Most of the sponsors take away the passports of the workers, who are mostly in the construction sector, for the duration of their contract.
"This system that is used to regulate the relationship between employers and migrant workers, with a work permit linked to a single employer, is problematic and a source of abuse against migrants," Crepeau added.
The official will present a comprehensive report of his visit to the UN Human Rights Council next June.
Hundreds of migrant workers have already died and many more injured on building sites in the country.
Labor unions have also warned that working conditions in Qatar could cause thousands of deaths before a ball is kicked.
Representatives of the families of the migrant workers -- killed or injured on building sites -- have called on FIFA to hand the tournament to another country, unless Doha can quickly guarantee workers' safety.