Measure to Improve: How Artificial Feedback Loops Accelerate Cyber Security Maturity

Written By Harry Brown, Cyber Security Consultant, Phronesis Security

The uneven learning environment across the cyber security industry, where some organisations learn painfully through direct experience while others remain in the dark about their actual security posture makes rapid improvement prohibitively difficult. Artificial feedback loops like the control effectiveness measurement techniques described in this article creates a clearer definition of success, enabling faster, more focused improvement.  

Feedback Loops Incentivise Progress 

In any complex system, feedback loops serve as the fundamental mechanism for improvement. They provide the necessary information to detect errors, measure progress, and adjust course accordingly. When feedback is immediate and frequent, learning accelerates, a principle that has transformed countless industries. 

Consider aviation's evolution in the 1970s-80s. During this period, the industry faced several major accidents stemming from human error, miscommunication, and procedural breakdowns in the cockpit. Like cyber incidents today, these accidents were rare but catastrophic. Initially, meaningful feedback came primarily through crash investigations; a reactive approach similar to how many organisations currently rely on data breaches to improve their cyber security posture. 

Aviation transformed itself by creating artificial feedback loops through simulation, near-miss reporting systems, and regular training scenarios that didn't require actual disasters to generate learning. This proactive approach dramatically improved safety, making commercial aviation one of the safest forms of transportation today. The cyber security industry stands at a similar inflection point, where relying solely on actual incidents for feedback is both inefficient and dangerous. 

This feedback principle mirrors the classic Plan-Do-Check-Act (PDCA) cycle, a cornerstone of continuous improvement methodologies. In cyber security, however, the 'Check' and 'Act' phases are often delayed or missing entirely due to the infrequent nature of real-world feedback. 

The Problem with Real-World Cyber Incident Feedback 

Cyber attacks present a particularly challenging feedback environment. Unlike systems where feedback is immediate and apparent, cyber incidents (the only true direct feedback mechanism) are both infrequent for individual organisations and often go undetected for extended periods. According to IBM's 2024 Cost of a Data Breach Report, the average time to identify and contain extortion attacks hovers around 290-300 days. For breaches involving stolen credentials, this timeline extends to 292 days. 

Even more concerning is the case of espionage-focused threat actors who aren't seeking immediate financial payouts. These sophisticated attackers can remain undetected for years within compromised systems, silently extracting valuable information while leaving minimal traces. 

This extended timeline creates a fundamental problem: feedback loops are most effective when they are short and frequent, allowing for rapid learning and iteration. The long and infrequent nature of cyber attack feedback prevents organisations from quickly learning from their mistakes and improving their defences. By the time an organisation discovers a breach, the attack techniques may have evolved, the security landscape may have changed significantly, and the lessons learned might already be outdated. 

IBM Cost of a Data Breach. Figure 31 

Further, organisations that haven't experienced a major breach lack direct feedback entirely, potentially leading to a false sense of security rather than improved practices. This creates an uneven learning environment across the industry, where some organisations learn painfully through direct experience while others remain in the dark about their actual security posture. 

Designing Synthetic Feedback Loops Around Control Effectiveness 

If we cannot rely on real incidents as our primary feedback mechanism, we must establish alternatives. Cyber risk remains the primary metric used to measure cyber security improvement, and control effectiveness is both a major input into risk calculations and the variable most within our control. All else being equal, increasing control effectiveness reduces residual risk. 

Therefore, by designing synthetic feedback loops to continuously measure and improve control effectiveness, we can systematically reduce residual cyber risk without waiting for actual incidents to occur. 

Several organisations have successfully implemented such synthetic feedback mechanisms. For example, traditional email filtering metrics like "number of blocked emails" provide limited insight. However, by applying more sophisticated techniques like F-Score measurements (which balance precision and recall), organisations can gain a much clearer picture of their email filtering effectiveness. This approach transforms a basic counting metric into a meaningful feedback loop that drives continuous improvement. 

For non-binary classification controls like vulnerability management, composite indicators can be designed to specifically track an organisation's goals over the next 1-2 years. Unlike trying to measure against broad controls descriptions taken directly from security frameworks, these custom indicators align directly with the organisation's specific objectives and for a given control. 

Multi-factor authentication (MFA) provides another example where nuanced metrics can create powerful feedback loops. By using weightings that account for the criticality or sensitivity of various systems and accounts, incorporating log data, and applying logarithmic scales that exponentially penalise metrics when single-factor authentication is used, organisations can create a much more accurate and actionable view of their authentication control effectiveness. 

For detection controls like Endpoint Detection and Response (EDR), organisations can either use simple F-Scores when focusing solely on the classification capability (malicious vs. non-malicious) or incorporate these scores into composite indices that include other variables like response time, providing a more holistic view of control performance. 

Even process-oriented controls can benefit from this approach. Metrics like "percentage of access revoked on time" not only measures current performance but often leads to automated solutions or process improvements that continuously enhance security posture. 

The Value of Consistent, Measurable Control Metrics 

Expert opinion plays an important role in assessing control effectiveness, but it often introduces variability and subjectivity. Without clear, repeatable measurement methods, feedback loops become ambiguous, making it difficult to define success or track improvement over time. 

A control effectiveness measure is only valid as a feedback loop if it gives the same result when applied by a network engineer or a philosopher. Consistency across bias and background is what makes it dependable. 

This consistency is critical. When control assessments vary based on who performs them, organisations cannot reliably determine if they're improving or merely experiencing evaluator variation. This ambiguity undermines the entire feedback process. 

Establishing consistent, measurable control metrics, even if they sacrifice some nuance, creates a clearer goalpost, enabling faster, more focused improvement. These metrics also serve to sustain team morale by showing tangible progress, which is often difficult to demonstrate in cyber security where success is typically defined by what doesn't happen (breaches, incidents, etc.). 

For example, vague control descriptions like "The organisation provides security awareness and training to employees to help them understand and fulfill their information security responsibilities" leaves too much room for interpretation and inconsistent assessment. 

In contrast, specific descriptions create measurable feedback loops: "All employees requesting administrator-level access to any system listed in Acme Inc's Critical Systems Register must complete the 'Secure Administration Training – Level 1' course (Course Code: SAT-L1) via the Acme Learning Management System (LMS). This training includes modules on access control configuration, secure remote access, logging and monitoring requirements, and incident reporting procedures. The training must be completed within 30 days prior to access being granted and must be repeated every 12 months. A minimum passing score of 85% on the end-of-course assessment is required. Training completion and assessment scores are recorded in the LMS and reviewed quarterly by the IAM team." 

The specificity of the second description allows for consistent measurement regardless of who performs the assessment, creating a reliable feedback loop that drives improvement over time. 

How to Get Started 

Infrequent and ambiguous feedback loops severely limit the cyber security field's ability to improve rapidly. By designing synthetic feedback mechanisms around measurable control effectiveness, organisations can create more consistent, morale-boosting, and risk-reducing learning environments. 

To implement effective synthetic feedback loops, organisations should: 

1. Start small and prioritise quality of measurement techniques over quantity. Focus on getting a few high-impact controls measured well and improved rather than attempting to measure everything at once. 

2. Understand that immediate automation should not be the primary goal. Many valuable data inputs require manual review or collection initially. For example, it's relatively easy to automatically determine how many applications are integrated with a central Identity and Access Management (IAM) source, but identifying the total number of applications (the denominator) often requires manual effort; yet this is where the true value lies. 

3. Get specific with control descriptions and KPIs. Vague controls cannot be consistently measured. The more specific and detailed the control description and effectiveness criteria, the more consistent its measurement will be across different assessors. 

4. Remember the philosopher test. As we develop control metrics, we should consistently ask: "Would a network engineer and a philosopher arrive at the same measurement result?" If the answer is yes, we're on the right track. 

By embracing these principles and designing synthetic feedback loops that don't rely on actual incidents, the cyber security industry can accelerate its improvement cycle, reduce residual risk, and create more resilient organisations without waiting for the painful lessons of real-world attacks. 

After all, in cyber security as in aviation, the most valuable lessons are those we don't have to learn the hard way.