The Wednesday decision came after a marathon seven-hour emergency session of the top UN human rights body, where Israelis and Palestinians traded accusations over war crimes.
The 47-member council backed a Palestinian-drafted resolution by 29 votes, with Arab and fellow Muslim countries joined by China and Russia, plus Latin American and African nations.
The United States was the sole member to vote against. The 17 abstentions were by the council's European members, plus Japan and South Korea.
The probe team, yet to be appointed, is tasked with reporting back to the council by March.
Israeli Prime Minister Benjamin Netanyahu's media office slammed the decision.
"This investigation by a kangaroo court is a foregone conclusion," his office claimed.
US ambassador Keith Harper alleged that the vote would undermine ceasefire efforts.
The session was called by Arab nations and fellow members of the Organization of the Islamic Conference.
The council's membership rotates, and Israel is not currently part of the UN body. Non-members cannot vote but are entitled to speak.
Israeli ambassador Eviator Manor lashed out at countries that piloted the vote.
NTJ/NJF