The plan is at the stage of examining how to send soldiers to help train Libyan security forces to fight terrorist taking over some region of the country.
SAS Forces
UK troops would be deployed in non-combat role, but SAS members would be armed to take out high value ISIS targets.
The security vacuum has enabled ISIS to expand its so-called caliphate to the country.
Under plans being looked at, British military personnel would contribute to a 6,000-strong Italian-led mission to train and support local forces.
The deployment would mean boots on the ground in an area where six SAS men were captured during an operation in 2011, months before the death of Libyan Dicatator Colonel Gaddafi.
Details of the mission came as Libya's opposing factions yesterday signed a UN-brokered deal to form a national unity government, in a move that western leaders hope will help in the fight against ISIS.
Four years after the fall of dictator Gaddafi, Libya remains deeply split with two rival governments – a self-declared one in Tripoli and an internationally recognized one in the east of the country.
Despite the signing, the heads of both rival parliaments and factions in both camps have rejected the agreement outright, according to Daily Mail.
Following its formation, Britain and its allies, thought to also include the United States and France, except to receive a request for assistance from the newly created Libyan administration.
Britain and Italy have been working with the European Union and the United Nations on a plan to support the new government by rebuilding the state and training security forces.
Special Forces could also be sent to the country to spearhead military operations targeting traffickers who are exploiting migrants wanting to make the perilous crossing to Europe.
The UK SAS Forces would work with specialist forces from other EU countries to help pinpoint the leaders of ruthless gangs, the vessels they are using the routes taken.
A report published this month revealed ISIS militants were building a 'retreat zone' in Libya to avoid air strikes in Syria and Iraq.
As many as 3,000 terrorists are believed to have travelled to the war-torn country to create a strategic hub for recruits unable to reach ISIS in its Syrian heartland.
The 24-page report, presented to the UN security council by a committee, concluded that ISIS views Libya as the 'best opportunity to expand its so-called caliphate'.
At present it numbers some 2,000-3,000 terrorists, with about 1,500 based in Sirte, Colonel Gaddafi's birthplace and the last major city to fall to rebels in 2011.
Their expanding influence over the region has become so concerning for Libyan leaders that they have announced they would accept help from the Russians.
The terrorist group is also said to be moving its leaders to the Libyan city of Sirte as a 'contingency' plan from where it can project across the whole of North Africa.
The shift in tactics also means the ISIS now has a foothold on the coast of Libya across the Mediterranean Sea from Italy.
A Downing Street spokesman said the Government had made clear it would support a new administration in Libya, but if any troops were sent they would not be in a combat role.
Foreign Secretary Philip Hammond told MPs on Wednesday the West and Persian Gulf states would 'swing behind' a new Libyan government to help them fight ISIS.