Gun shots were heard near the area shortly after the explosion and ambulances rushed to the scene, a Reuters witness said.
Several army vehicles also rushed to the scene in the old district of the capital Sanaa which is also home to the country's central bank, the witness said.
"There are dead and wounded," a security source told AFP without giving any immediate figures.
Violence is common in Yemen, where an interim government is grappling with al Qaeda-linked militants as well as severe economic problems inherited from veteran President Ali Abdullah Saleh who was forced out of office in 2011.
"The attack took place shortly after working hours started at the ministry when a suicide bomber drove a car into the gate," the Defense Ministry source said.
The exact number of casualties was still unclear.
"The explosion was very violent, the whole place shook because of it and plumes of smoke rose from the building," an employee who works in a nearby building told Reuters.
NJF/NJF