The controversial new laws, laid before Parliament in September, give public authorities – other than the police and intelligence agencies – access to personal data held by telcos and ISPs for periods of up to twelve months.
They form part of a package of measures produced under the Anti-Terrorism, Crime & Security Act (ATCSA), which was enacted in the aftermath of the September 11th atrocities.
This Act required the retention of communications data on the grounds that these were needed for the purpose of fighting terrorism. These data, retained on anti-terrorism grounds, will be accessible by provisions under the Regulation of Investigatory Powers Act (RIPA) to a range of public authorities for purposes unconnected with terrorism.
Communications data are those data, retained by telcos (including ISPs and interactive television service providers), which describe the caller and the means of communication (e.g. subscriber details, billing data, e-mail logs, personal details of customers, and records showing the location where mobile phone calls were made). Communications data do not contain the content of the communications.
Critics, including the Parliamentary Joint Committee on Human Rights, are concerned that the measures will breach the European Convention on Human Rights, and until late on Thursday it looked likely that Conservative, Liberal Democrat and Cross Bench peers would oppose the Orders.
However, at Government ultimately secured its victory.
A 'fatality' motion by Deputy Opposition leader Baroness Blatch provoked calculated outrage on the government benches. Her motion, which proposed to kill the government's Order outright, was a device that had not succeeded in the Lords in 30 years.
Government members threatened that the very basis of the House would fundamentally change if the fatality motion was passed. The Government vowed that if the motion succeeded it would exact revenge by killing any and every Order proposed by a future Conservative government.
The Liberal Democrats, which until then had agreed to block the data retention Orders, reversed its position. By 4pm, faced with certain defeat, the Conservatives were then forced to withdraw the motions. The proposals passed unopposed.
This means that after a short period of implementation of a much derided 'voluntary' retention scheme, the Government will be able to pass regulations forcing all communications providers to store and yield information on all their customers.
However, two additional motions, favoured by activist groups Privacy International and the Foundation for Information Policy Research, were approved by the Lords.
The first, proposed by Lord Phillips of Sudbury, states that the Interception of Communications Commissioner should be compelled to inform "any person who appears to have been adversely affected by any wilful or reckless failure on the part of any person exercising or undertaking any of the powers and duties conferred or imposed on him by the Regulation of Investigatory Powers Act 2000".
This means that, for the first time, the oversight body must let people know when their privacy has been improperly invaded.
Baroness Blatch also succeeded in a motion requiring the government to report to Parliament the extent of overseas access to personal information stored by communications providers. Such access is widespread, and available to many developed and developing countries.
A furious Baroness Blatch also put the House on notice that a Private Members Bill would be tabled next year to redress many of the problems identified by the Opposition.
Privacy International called the passing of the Orders "a dark moment in the history of the House of Lords."
"The government should never have succeeded," commented Simon Davies, Director of Privacy International. "The proposals were approved only because the Liberal Democrats caved in under pressure from unprecedented threats by the government".
Mr Davies vowed to continue fighting the proposals, in the courts and through the use of more aggressive campaigning tactics. "I would have no hesitation is taking legal action against Communications Service Providers who comply with a regulation that is unlawful, regressive, dangerous and unnecessary," he said.
The text of the House of Lords debate is available through Hansard.