BBC -- In an effort to stem migration, the EU is actively supporting a "system of abuse and exploitation" on Libyan shores, the group said in a report.
EU funds are going to authorities working with militias and people smugglers, the report says.
The EU has provided ships, training and funding to the Libyan coastguard.
Libya is the main thoroughfare for migrants trying to reach Europe. Arrivals in Italy - the main destination for boat crossings - fell sharply following the provision of EU funds to the Libyan coastguard.
But Amnesty says the coastguard is working with criminal gangs and people smugglers who are guilty of a range of abuses, with the knowledge of EU officials.
It alleges that the drive to stem migration has resulted in "mass, arbitrary and indefinite detention" of refugees and migrants.
Refugees and migrants intercepted by the Libyan Coast Guard are sent to detention centres run by Libya's general directorate for combating illegal migration (DCIM).
Amnesty said about 20,000 people were currently detained at these centres, and it had testimony showing detainees were subject to "torture, forced labour, extortion, and unlawful killings" at the hands of authorities, traffickers, and militias.
"Tens of thousands are kept indefinitely in overcrowded detention centres where they are subjected to systematic abuse," said John Dalhuisen, Amnesty's Europe Director.
"European governments have not just been fully aware of these abuses; by actively supporting the Libyan authorities in stopping sea crossings and containing people in Libya, they are complicit in these abuses."
When a BBC correspondent visited a detention centre in earlier this year, detainees described violence and abuse and conditions "like hell" and "even worse than jail".
Speaking to Amnesty, one man from the Gambia who was detained for three months said he was starved and beaten.
"They beat me with a rubber hose, because they want money to release me. They call the family while beating [you] so the family send money," he said.
The human rights group said it believed it had enough evidence to take EU governments to court.
Libya has spiraled into lawlessness since Nato-backed forces overthrew long-serving ruler Colonel Muammar Gaddafi in October 2011, with a myriad armed militias vying for control.