The phone-tapping admission of Clive Goodman, royal editor of the News of the World, has not done much for the image of investigative journalism. His downfall follows repeated exposures of the murky ways in which Goodman's NoW colleague, the "fake sheikh" Mazher Mahmood, sets out to entrap his victims, such as those behind the so-called plot to kidnap Victoria Beckham. Rebekah Wade, editor of the Sun, also once famously admitted paying police officers for information. The Sunday Times, too, has been accused of using illegally obtained tax records for a story about Tory treasurer Michael Ashcroft, and of making off with sheaves of Cabinet Office correspondence purloined by a temporary secretary.
All these newspapers belong to Rupert Murdoch, whose editors seem to play dirtier than most. But there is not a newspaper or TV channel in the country that has not, on occasion, got down in the gutter and used questionable methods.
Investigative journalism is not a dinner party, particularly in a secretive country like ours where the privacy cards are stacked in favour of the rich and powerful. But it all depends on what the target is.
I've used some of those questionable methods myself over the years. I, too, once listened to the mobile phone messages of a corrupt arms company executive - the crime similar to that for which Goodman now faces the prospect of jail. The trick was a simple one: the businessman in question had inadvertently left his pin code on a print-out and all that was needed was to dial straight into his voicemail.
There is certainly a voyeuristic thrill in hearing another person's private messages. But unlike Goodman, I was not interested in witless tittle-tattle about the royal family. I was looking for evidence of bribery and corruption. And unlike the News of the World, I was not paying a private detective to routinely help me with circulation-boosting snippets. That is my defence, when I try to explain newspaper methods to my current university journalism students, and some of whom are rather shocked. There are other techniques I have used, along with the rest of Fleet Street. I did not turn up my nose when the notorious Benjy the Binman emptied a bag of stinking rubbish on to my carpet. He wanted to show me incriminating statements about Saudi arms deals, which a City law firm had been too idle to shred before putting out on the street for collection. I read the information with interest. I did, however, refuse to pick up the other gossipy documents about celebrities that Benjy was also peddling. And when he wanted large amounts of cash for copies of those documents he had that were rather more in the public interest, I sent him off to the Sunday Times.
Then there is the question of "stings". Hidden cameras have developed technically at great speed in recent years. They have changed from bulky items toted around in hold-alls, to tiny implants in spectacle-frames. As a result, undercover investigations became fashionable. Too fashionable, I think. The original "undesirable journalist", the German reporter Günter Wallraff, disguised himself as a Turkish guest-worker in the 1980s, to expose real discrimination and exploitation. More recently, it seems as though the telegenic Donal MacIntyre started a trend for going undercover to expose such predictable targets as football hooligans, the fashion industry and the BNP, just because he could.
But we all use deception. I still treasure the moment when I rang up Mark Thatcher in Downing Street. Thatcher was secretly on the payroll of a firm trying to get a construction deal in Oman. But at the time, we could not yet prove a link between him and the Middle East fixer concerned, whose name was Jamil Amyuni. "Who's calling?" said the Downing Street switchboard. I said "Tell him it's Jamil Amyuni". In two seconds flat, Mark came on the line, and shouted cheerily "Hi, Jamil!" We had our story. Was I wrong to do that? Surely not. We were successfully exposing what many people thought was misbehaviour by the then prime minister's son, who was shamelessly exploiting his position.
I think the rule should be that deceptions, lies and stings should only be used as a last resort, and only when it is clearly in the public interest. And, as for actually breaking the law? Well, it is hard to keep on the right side of legality on all occasions. Like most investigative journalists, I have had my share of confidence injunctions, lost libel actions and threats of prosecutions under the Official Secrets Act.
These tend to breed disrespect for the law, and a nonchalant attitude to these billionaires and cabinet ministers who wheel in solicitors when it suits them, to try to conceal their own crimes and misdemeanours. When we revealed the truth about corrupt Conservative minister Jonathan Aitken, he called a televised press conference to issue a writ and describe me as a cancer cell, practising "bent and twisted journalism". That sort of experience gives an investigative reporter a thick skin.
But clearly the present black market in what privacy campaigners call "data rape" has got out of hand. Goodman's conviction comes at a pivotal moment. What he did has turned out already to be a serious crime under the Regulation of Investigatory Powers Act.
But Richard Thomas, the information commissioner, is lobbying for deterrent jail sentences, for the related, and widespread, technique of "blagging". He is bringing a spate of prosecutions against the private eyes who some journalists regularly use as cut-outs to make pretext phone calls to banks, tax authorities and call-centres.
Thomas says there is a public interest defence available under the Data Protection Act and honest journalists have nothing to fear. We shall have to see about that. Personally, I am resigned to seeing the tabloid cockroaches doused with a spot of legal insecticide. Driven by greedy and cynical proprietors, and making no distinction between gossipy intrusions and genuine public interest investigations, they are bringing our trade into disrepute.
· David Leigh is the Guardian's investigations editor and professor of reporting at City University
• This article was amended on 5 August 2011. The original – while attributed in the newspaper and online to David Leigh – also carried an online credit for Ian Reeves. This has been corrected.