Audit Analytics: Is sampling enough for internal controls testing?

Sampling theory

Traditionally, sampling would be used to test internal control effectiveness, using sample size guidance, usually something like this:

• For a control that operates monthly, test 2
• For a control that operates weekly, test 8
• For a control that operates daily, test 30
• For a control that operates more than once a day, test 30

The logic behind this guidance is that if the control has operated effectively for the sampled instances, you can be comfortable that it has operated effectively every time.

Sampling reality

Unfortunately, in reality this is often not the case – as auditors we see it time and time again in well-known, large organizations, which appear to have strong controls environments. They get a case of the “This one’s” or “This time’s”:

1. All “Disbursements over limit” are meant to be approved by two directors but “this one” was incredibly urgent so only one signature was provided
2. Inventory counts for high value goods should be conducted weekly, but, “this time” they were not done for 2 weeks while the responsible person was out of office
3. Purchase orders should to be created and approved by 2 different people – “this one” managed to be released to suppliers without approval or approved by the creator of the PO
4. The Finance Director is not supposed to be able to post journals, but “this time” their system access has not been setup correctly, so they can
5. Purchase invoices over $100,000 should be approved by the CEO, but “this one” was only just over so was just signed by the CFO

When occurrences like these are identified using sampling with a few items, it is clear that there may be many, many others that are missed. Of even more concern, when using sampling, you potentially have a very high likelihood of missing these single control failures. Consider:

• If you only test 2 examples of a monthly control, there is an 83% chance you’ll miss a single control failure during the year
• If you test 8 examples of a weekly control, there is an 85% chance you’ll miss a single control failure
• If you test 30 examples of a daily control (assuming it is carried out on weekdays), there is an 88% chance you’ll miss a single control failure
• If you test 30 examples of a control that operates more than once a day, there could be a 99% chance (or higher) that you’ll miss a single failure

A solution

The only way to get any real comfort over the operating effectiveness of an internal control procedure is to test every instance of it running.

Now, if the control is conducted completely offline, outside of any systems, that is likely to be impractical. However most controls are now evidenced somewhere in an organization’s computer systems. By obtaining a download of transactions from these systems, you can very easily test the control against 100% of the transactions, using Computer Aided Audit Tools (CAATs; also known as “audit data analysis” or “data analytics”).

As an example, let’s look at what can happen to the “This one’s” and “This time’s” from our 5 sampling scenarios above when we utilize a data analytics tool:

1. Easily extract all approved transactions with less than two approvers. For those with two approvers you can match those approvers to a list of management/directors to ensure they all have approval authority and the limits
   Watch Video
2. Summarize the inventory count log by week and location to ensure that all locations were counted at least once a week
   Watch Video
3. Use an exception report to extract all entries where the creator and approver are the same or where there is no approval at all
   Watch Video
4. Match the journal transactions to the approved system access log to identify all transactions entered by non-approved personnel
   Watch Video
5. Extract all purchase invoices over $100,000 where the approvers do not include the CEO
   Watch Video

With a data analytics tool, all of these analytics can be conducted in a matter of minutes. And, each of these tests can be made repeatable to easily run on a regular basis, or, if need be, by different auditors. These are just a few simple examples, but data analysis can be used in many different ways to test a wide range of internal controls.