Majed Matir was leaving his home in Qotn in the province of Hadramawt when two men on a motorcycle opened fire killing him, the ministry said on its website 26Sep.net on Sunday.
Matir led the special forces unit in the city that was responsible for fighting terrorism and countering riots.
The attack was probably carried out by members of al-Qaeda, the ministry said citing a security official.
Al-Qaeda, which is well established in the southern and southeastern regions of Yemen, has carried out multiple attacks against security officials who are chasing the network's members.
Around 70 security officers have been killed in two years in this part of Yemen in attacks attributed to al-Qaeda, officials say.
On Friday, the Yemeni interior ministry said that Ansar al-Sharia, a group linked to al-Qaeda, tried to capture a town in Hadramawt.
Al-Qaeda fighters have escaped to mountainous areas after being forced out of urban centers in southern Yemen following an army offensive in May 2012.
Al-Qaeda has benefitted from the weakening of the central government in 2011 after a popular uprising against former president Ali Abdullah Saleh.