Why headline security scores can hide real progress

The central problem

A simple number can conceal a complicated result

A percentage rises. A risk score falls. A gauge moves from red towards green. For security teams, managers and boards, that simplicity is useful: complex environments need a way to communicate direction.

Risk begins when the headline number becomes the outcome rather than an indicator.

Example 1 • Active Directory

The 100/100 score that did not move

Two PingCastle Active Directory health assessments were compared roughly three months apart. At headline level, the result looked disappointing:

PingCastle's Domain Risk Level uses the highest of its main indicator scores. Stale Objects, Privileged Accounts and Anomalies all remained at 100/100, so the overall result correctly continued to signal significant residual Active Directory risk.

But the rule-level evidence told a different—and equally important—part of the story.

The KRBTGT account had previously retained a password dating from August 2016. The follow-up evidence showed two controlled resets, and the associated Golden Ticket rule no longer matched. Certificate-related exposure also reduced substantially.

Important attack paths have been reduced, but substantial residual risk remains.

That conclusion is more useful than either “nothing improved” or “the remaining 100 can be ignored”.

Example 2 • Zero Trust

A 2.8-point improvement that also hid the detail

The overall Microsoft Zero Trust Assessment increased from 40.7% to 43.5%. Viewed alone, that could suggest modest progress.

Control state mattered too. Several Conditional Access policies remained in report-only mode so impact could be evaluated before enforcement. That was evidence of progress towards a control—not evidence that a preventive control was already enforced.

Keep the tool—improve the interpretation

The dashboard is not the problem

Microsoft Secure Score, Zero Trust assessments, vulnerability scanners and Active Directory assessment platforms can expose weaknesses, create repeatable checks, identify priorities and make technical information easier to communicate. The research behind RACF-CC relied extensively on them.

The problem begins when an organisation allows the score to replace the analysis.

Movement in a score and movement in risk are related, but they are not necessarily the same thing.

A stronger reporting pattern

Use the score as one layer of evidence

1. 01

   Start with the headline indicator

   Use it to understand broad posture and identify areas requiring investigation.

2. 02

   Examine the controls underneath it

   Determine which individual findings improved, worsened or remained unresolved.

3. 03

   Verify the implementation

   Use configuration, policy state, logs, vulnerability results or other direct evidence to establish what changed.

4. 04

   Relate the change to risk

   Identify which credible attack path, exposure or operational consequence the control affects.

5. 05

   Record what remains

   Keep residual risk, exceptions and controls still in pilot, audit or report-only states visible.

Example 3 • Vulnerability management

Before comparing two numbers, confirm they measure the same thing

A scanner may report more critical findings because security deteriorated. It may also have authenticated successfully to more hosts, brought additional subnets into scope, reached previously unavailable assets or changed classification rules.

Raw Nessus totals in the RACF-CC research were treated cautiously because host availability and credentialed visibility changed between assessments. The more defensible comparison used 84 server-network hosts visible in both datasets.

Measurement over assumption

Evidence-led does not mean metric-led

Being evidence-led means supporting a security claim with appropriate evidence—not collecting the largest possible number of metrics or allowing dashboards to dictate risk decisions.

The measure should follow the security question—not the other way around.

Resource-aware decisions

Why this matters particularly for small teams

Resource-constrained organisations cannot remediate everything simultaneously. A team may spend significant time resolving one high-impact identity weakness while dozens of lower-priority dashboard recommendations remain visible and the overall score barely moves.

That does not necessarily make the work a poor use of time. The relevant questions are whether the action materially reduced a credible risk, whether it was implemented safely, and whether the improvement can be demonstrated.

This is why RACF-CC separates expected security benefit from implementation effort, operational impact and cost. The purpose is not to optimise a dashboard score. It is to decide what deserves attention.

The practical conclusion

Scores should start conversations, not finish them

Aggregate assessments remain valuable for discovering gaps and understanding direction. But the strongest evaluation evidence usually sits underneath the headline: before-and-after control metrics, configuration evidence, comparable vulnerability data, rule-level changes and operational validation.

A PingCastle Domain Risk Level of 100/100 still communicated that important Active Directory risks remained. A Zero Trust score in the low forties still showed that substantial work was unfinished. The mistake would be interpreting either number as a complete description of what had—or had not—changed.

What changed? Which attack path was reduced? Is the control actually enforced? What evidence proves it? What residual risk remains?

Headline security scores help us navigate. They should not become the destination.

From research to RACF-CC

Carry the evidence into the next decision

This finding emerged from the postgraduate research that developed the original NIST Lite framework and later informed RACF-CC. The Evidence stage carries the same principle forward: record what was implemented, retain appropriate evidence, evaluate whether the intended control is operating and keep residual risk visible.

Evidence note: Figures in this article are anonymised evaluation results from the dissertation underpinning RACF-CC. Aggregate results are presented with their stated comparability and enforcement limitations; they are not certification claims.