Hm.. not quite sure of the rules yet - but did the query I posted earlier give incorrect results for Application 1 (but all else seemed ok)? if so, can you try this?
SELECT a.Application, a.N, b.N FROM
( SELECT Application, COUNT(DISTINCT CacheId) N FROM Audit a WHERE Source = 'direct' GROUP BY Application ) a
( SELECT Application, COUNT(DISTINCT CacheId) N FROM Audit a WHERE Source = 'Cache'
AND NOT EXISTS (SELECT * FROM Audit b WHERE b.Source = 'direct'
AND a.Application = b.Application AND a.cacheid = b.cacheid )GROUP BY Application
) b ON a.Application = b.Application;