By Jason Hoenich on May 11, 2018
Metrics don’t just paint a picture of the past; they provide a pathway to the future. Metrics done right can not only identify areas of training that require improvement but also identify employees who need additional help.
In addition to providing ROI for information security expenditures, metrics are instrumental in:
- Identifying major data breaches and other vulnerabilities
- Attracting high-quality security personnel
- Increasing customer trust and loyalty
- Ensuring compliance with legal and self-regulatory frameworks such as PCI DSS, DMCA, and HIPAA
Why are metrics not being done?
Collecting metrics in a constantly changing risk environment can be challenging, especially given the lack of universally accepted measurements. Many enterprises with security awareness programs don’t collect metrics at all.
An absence of metrics can be especially problematic for small-to-medium size businesses because they may lack a Learning Management System (LMS), which some organizations use for collecting data. An absence of metrics means no realistic ROI can be supplied, which may make executives reluctant to commit resources for an integrated security awareness program. Because haphazard efforts make documentation even more difficult, this leads to a vicious circle of a lack of data leading to no ROI, which leads to inadequate resources, which leads to no structured program, and so on.
However, the biggest single barrier to collecting meaningful metrics may be the difficulty of measuring actions. Does the security awareness program actually create more responsible behavior by the employees? Metrics that note the employees’ participation in the security program’s initiatives must be accompanied by those that describe improvements in their actions. Measuring input is easy; measuring output is hard.
The good news is that there are straightforward ways of obtaining accurate, useful metrics. Here are five ways of securing important metrics.
1. Phishing Training
Phishing training is a relatively easy way of obtaining metrics. Before the training, establish a baseline by recording the number of employees who fall prey to social engineering attacks, as well as how many file reports on suspicious emails. After training, run a test campaign using phony phishing emails. Keep in mind this is to test how well the employees absorbed the training, so try to vary the tactics and times of the fake emails. By comparing the behavior of the employees before and after the training, you can measure its effectiveness. Testing also has the benefit of engaging the employees and reinforcing their training.
Beware of these common mistakes in phishing training:
- Do not rely only on phishing metrics for your security awareness program.
- Don’t ignore the wealth of information available in the test campaign. For example, record changes in the time from incident to detection. Identify effective phishing tactics, as well as the most vulnerable employees.
- Test everyone. Do not assume that security personnel or high-level executives are immune.
- Remember that the goal is to change behavior, not punish. Refer to employees who frequently fall prey to fraudulent emails as “repeat responders,” not “repeat offenders.”
Annual surveys provide significant benchmarks for the staff’s attitudes toward information security, as well as their understanding of organizational policies. The effectiveness of security awareness training can be demonstrated by comparing the responses on one year’s survey to another. If many employees display ignorance about a security problem, training can be adjusted.
Surveys also reinforce the security awareness training. For example, the question, “Do you know that you are accountable if someone else uses your workstation for illegal purposes?” reminds employees to lock their workstations when leaving for the night.
3. Behavior Change Metrics
As mentioned above, the real trick to measuring the effectiveness of a security awareness program is tracking behavior. Key behavioral metrics include:
- Amount of reported lost or stolen devices
- Increase in phishing email reports
- Decrease in reaction time of incident response teams to reported phishing emails
- Hours spent by staff learning at voluntary events
4. Videos training
People love videos, especially ones with strong production values. Video-influenced behavior changes can be measured with before and after benchmarks, similar to the phishing testing described above. Engagement can be measured by the number of views, time spent viewing, and shares.
5. Live Trainings
Communication at a live training should be a two-way street. Are the employees’ questions becoming more sophisticated over time? Are there some security areas that they are resistant to understanding, even after repeated trainings? Pay attention to what your employees are saying.
These are only a few ways of measuring the effectiveness of your security awareness program. Your company may need a tailored program for its specific requirements.
For more information on how metrics can help your current security awareness program succeed, join us on Wednesday, May 23rd, at 11 am PST/2 pm EST for our fireside chat on Security Awareness Metrics in our third installment of Habitu8's Mentoring Series: 5 Key Metrics For Building Security Awareness Programs.