"No contact has been made" with the gunman, who appeared to be acting alone, New South Wales police commissioner Andrew Scipione told a press conference.
Three people were seen leaving the building. One man wearing an apron left through the rear fire door saying "don't shoot," while two people left through the front door, ABC news reported.
It was not clear whether the ones who left from the front had been in the cafe with the hostages, the BBC reported. All were led away by police.
Police were "not tagging this as a terrorist incident" at this stage, he said, but were exploring all possibilities.
Prime Minister Tony Abbott said "obviously there are some indications that it could be" politically motivated.
Two apparent hostages earlier held a black flag up to the window of the Lindt Cafe in Sydney's Martin Place, bearing the Arabic phrase in white Arabic script like ISIS flag.
"Australia is a peaceful, open, and generous society. Nothing should ever change that," Abbott said. "That's why I would urge all Australians today to go about their business as usual."
Footage on several news channels showed a middle-aged man with short silvery beard and a black bandanna on his head with Arabic writing on it, speaking and gesturing to people in the cafe.
He was wearing a black backpack, a white t-shirt and a black vest.
The area was cordonned off and the state parliament at the end of Martin Place was in lockdown with staff told to stay in their office and lock the doors.
The US consulate, 50 metres from the cafe, was closed and evacuated.
A park in the city and the Sydney Opera House was also evacuated after suspicious packages were reported.
Flights were diverted from over the central business district, with reports saying the move was to allow security forces to operate helicopters if needed.
Many business closed early to allow staff to get home amid the disruption.
Lindt Australia chief executive Steve Loane told news.com.au he believed there were 40 to 50 people inside the building including customers and staff.
Police earlier Monday arrested a 25-year-old man in western Sydney for allegedly plotting an attack in Australia and helping Australian citizens travel to Syria, ABC reported.