Category: Information Risk Management

On controls and their systemic ineffectiveness per se. As written about a lot in the past year on this site, PCAOB now finally seems to find out how things have been ever since SOx… in [simple block quote copy from this post by James R. (Jim) Peterson]:

The PCAOB Asks the Auditors an Unanswerable Question: Do Company Controls “Work”?

“Measure twice – cut once.”
— Quality control maxim of carpenters and woodworkers

If there can be a fifty-million-euro laughingstock, it must be Guillaume Pepy, the poor head of the SNCF, the French railway system, who was obliged on May 21, 2014, to fess up to the problem with its € 15 billion order for 1860 new trains—the discovery after their fabrication that the upgraded models were a few critical centimeters too wide to pass through many of the country’s train platforms.

Owing evidently to unchecked reliance on the width specifications for recent installations, rather than actual measurement of the thirteen hundred older and narrower platforms, the error is under contrite remediation through the nation-wide task of grinding down the old platform edges.

That would be the good news – the bad being that since the nasty and thankless fix is doubtless falling to the great cohort of under-utilized public workers who so burden the sickly French economy, correction of the SNCF’s buffoonish error will do nothing by way of new job creation to reduce the nation’s grinding rate of unemployment.

The whole fiasco raises the compelling question for performance quality evaluation and control – “How can you hope to improve, if you’re unable to tell whether you’re good or not?”

This very question is being reprised in Washington, where the American audit regulator, the Public Company Accounting Oversight Board, is grilling the auditors of large public companies over their obligations to assess the internal financial reporting controls of their audit clients.

As quoted on May 20 in a speech to Compliance Week 2014, PCAOB member Jay Hanson – while conceding that the audit firms have made progress in identifying and testing client controls — pressed a remaining issue: how well the auditors “assess whether the control operated at a level of precision that would detect a material misstatement…. Effectively, the question is ‘does the control work?’ That’s a tough question to answer.”

So framed, the question is more than “tough.” It is fundamentally unanswerable – presenting an existential problem and, unless revised, having potential for on-going regulatory mischief if enforced in those terms by the agency staff.

That’s because whether a control actually “works” or not can only be referable to the past, and cannot speak to future conditions that may well be different. That is, no matter how effectively fit for purpose any control may have appeared, over any length of time, any assertion about its future function is at best contingent: perhaps owing as much to luck as to design — simply not being designed for evolved future conditions — or perhaps not yet having incurred the systemic stresses that would defeat it.

Examples are both legion and unsettling:

• The safety measures on the Titanic were thought to represent both the best of marine engineering and full compliance with all applicable regulations, right up to the iceberg encounter.
• A recovering alcoholic or a dieter may be observably controlled, under disciplined compliance with the meeting schedule of AA or WeightWatchers – but the observation is always subject to a possible shock or temptation that would hurl him off the wagon, however long his ride.
• The blithe users of the Value-At-Risk models, for the portfolios of collateralized sub-prime mortgage derivatives that fueled the financial spiral of 2007-2008, scorned the notion of dysfunctional controls – nowhere better displayed than by the feckless Chuck Prince of Citibank, who said in July 2007 that, “As long as the music is playing, you’ve got to get up and dance… We’re still dancing.”
• Most recently, nothing in the intensity of the risk management oversight and reams of box-ticking at Bank of America proved satisfactory to prevent the capital requirement mis-calculation in April 2014 that inflicted a regulatory shortfall of $ 4 billion.

Hanson is in a position to continue his record of seeking improved thinking at the PCAOB — quite rightly calling out his own agency, for example, on the ambiguous and unhelpful nature of its definition of “audit failure.”

One challenge for Hanson and his PCAOB colleagues on the measurement of control effectiveness, then, would be the mis-leading temptation to rely on “input” measures to reach a conclusion on effectiveness:

• To the contrary, claimed success in crime-fighting is not validated by the number of additional police officers deployed to the streets.
• Nor is air travel safety appropriately measured by the number of passengers screened or pen-knives confiscated.
• Neither will any number of auditor observations of past company performance support a conclusive determination that a given control system will be robust under future conditions.

So while Hanson credits the audit firms – “They’ve all made good progress in identifying the problem” — he goes too far with the chastisement that “closing the loop on it is something many firms are struggling with.”

Well they would struggle – because they’re not dealing with a “loop.” Instead it’s an endless road to an unknown future. Realistic re-calibration is in order of the extent to which the auditors can point the way.

And … there you go, for today’s sake:

[Watching (us against) you …]