Simulated Phishing Training — What Changes When You Ring a Fake Doorbell?

"This Email Was a Training Exercise"

One morning, a plausible-looking work email arrives in your company inbox. When you click the link inside, an announcement appears instead of the expected screen. The notice explains that the email you just clicked was a fake training email sent by the company, and points out the suspicious details that were missed in it. A few weeks later, a similar email arrives again.

Phishing simulation training is an exercise where organizations send realistic-looking fake phishing emails to employees without prior notice, to see who falls for it and to educate them. Phishing here is a trick that uses messages disguised as coming from trusted sources to extract information like passwords or to make people open harmful links and files ("Phishing and Social Engineering — Ringing the Doorbell Instead of Breaking the Lock").

Behind this training lies a widely held belief: that teaching employees and practicing with fake emails makes them less likely to be fooled. It is a plausible belief. Because phishing emails are massive in volume and designed to deceive in the first place, filtering them out with technology has limits, and human judgment remains the final line of defense (PMID 33078362). If the hand that opens the door is inside the house, it seems natural to practice with that hand by ringing the doorbell falsely.

"Phishing and Social Engineering" stopped at the point where the results of this training are mixed. This article asks what comes next. It worked in some places and not in others, and where did that difference come from?

The More It Was Repeated, the Fewer People Clicked

Six American healthcare institutions sent 2,971,945 simulated phishing emails to their employees between 2011 and 2018, and 422,062 of those were clicked. That is 14.2%, or about one in every seven emails (PMID 30848810). The click rate is the proportion of sent simulated phishing emails in which a link was clicked. The median click rate for campaigns varied widely across institutions, ranging from 7.4% to 30.7% (PMID 30848810). This means that even for the same named training, the numbers differed by more than fourfold depending on the institution.

In this data, the more times an institution repeated the simulated campaigns, the lower the likelihood that emails in subsequent campaigns would be clicked. It was lower in the 6th to 10th campaigns, and even lower past the 10th. The researchers noted that this suggests a possible benefit of simulation training and awareness education (PMID 30848810). However, this study retrospectively collected already accumulated records, and the sample consisted of institutions from which data could be obtained. It merely shows that an increase in campaigns went hand in hand with a decrease in clicks, rather than separating the cause through an experiment.

The same direction was observed within a single healthcare institution. While 5,416 employees received all twenty simulated emails, the click rate decreased across all groups, regardless of whether they were frequent clickers or not (PMID 30861069).

There are also field experiments comparing different methods against each other. In an experiment involving over 10,000 employees at a Dutch government ministry, informational warnings about risk and hands-on experiences with simulated phishing were tested separately and together. Both methods significantly reduced the proportion of employees who handed over their passwords, and combining the two did not yield a larger effect (PMID 31851688). The outcome measured in this experiment is the proportion of employees who handed over their passwords. The informational warnings resemble security awareness education, which is education and training designed to teach people to recognize and cope with deceptions.

In settings where simulated phishing was repeated, fewer people clicked, and warning about or experiencing risks reduced the employees who handed over passwords. The first half of the conventional wisdom has evidence up to this point.

Teaching at the Exact Moment of Clicking

In the morning scene at the very beginning, the notice that appears when you click the link has a name. Embedded training is a method that shows a training screen at the exact moment someone clicks a link in a simulated phishing email. It stems from the idea that the moment you are fooled is the best time to learn.

In experiments by the researchers who created this method, the results were good. In a laboratory setting, embedded training was more effective than the conventional practice of sending security notices (DOI: 10.1145/1240624.1240760). People learned better when the same educational material was shown immediately after being tricked rather than sent separately by email, retained what they learned longer, and applied it better to other types of phishing emails (DOI: 10.1145/1299015.1299022).

Outside the laboratory, the results held up at first. In a field trial applied to employees at a company in Portugal, trained people were noticeably less fooled by a simulated attack a week later. There were even results suggesting that even colleagues who did not directly receive the training messages appeared trained (DOI: 10.1109/ecrime.2008.4696970). In a real-world trial involving 515 participants, the learned content remained even after 28 days, and adding a second training message reduced the rate of giving information to fake sites (DOI: 10.1145/1572532.1572536). Both trials are from the same development research team.

In other fields, the results were different. In a real phishing trial targeting 191 employees at a company in Italy, employees felt that the embedded training was effective, but the training did not reduce the extent to which they were fooled by phishing (DOI: 10.1145/3407023.3409179). In a large-scale experiment comparing four phrasing styles of training messages while repeatedly sending customized emails to employees, the phrasing styles made no noticeable difference in the likelihood of clicking the next email. Many people either clicked all links or none at all, regardless of whether they received training (DOI: 10.1109/msp.2013.106).

One study that tested embedded training on a large scale and over a long period is an experiment conducted with a single company. Over 14,000 employees received multiple simulated phishing emails during their routine work over 15 months. In the authors' words, embedded training used just like the way commonly found in today's industry did not make employees more resilient to phishing, and could instead produce an unexpected side effect of making employees easier to fool (DOI: 10.1109/sp46214.2022.9833766). Why that side effect occurs is not stated in the abstract of this study.

The method of teaching at the moment of clicking worked in experiments and field trials by the researchers who created it, but showed no clear effect in field trials by other researchers, and in a study following over 14,000 employees for 15 months, this training used just the way it is commonly used in the industry did not make employees more resilient.

What Lingers and What Fades

Twenty-eight days, a week. The earlier success cases confirmed their effects mostly within a few weeks after training. Effect decay is the phenomenon where the effects seen right after training fade as time passes.

In a review examining 42 studies on email phishing training, the short-term effects of training were well documented, but evidence on whether behavior changes for the long term was sparse (DOI: 10.1016/j.cose.2023.103695). Email habits also appear to be related to this decay. There is evidence that as people soon return to their usual email usage patterns, the training effect fades within a short period (DOI: 10.1177/0093650215627483).

When effects fade, a way forward is to attach support right at the point of decision-making, instead of relying on memory. In an online experiment, the group that included short training inside the warning screen better caught fake login screens than the group shown only the conventional browser warning, both in the next step without a warning and in the step one week later (PMID 30526089). In an experiment with college-aged adults, a classification aid that constantly displayed the characteristics of phishing emails nearby improved detection, and the effect was greatest when that aid was placed inside the email interface. These researchers noted that the outcomes of traditional methods involving brief, one-time instruction were limited (PMID 35157483). Both studies are results measured through experimental tasks, not results from long-term use in actual organizations.

Technical devices in the field point in the same direction. In a 15-month study, warnings attached to emails worked well (DOI: 10.1109/sp46214.2022.9833766).

There is contrary evidence on a timescale of several months. In a field study tracking 420 people in an organization for six months, positive behavioral changes appeared statistically clear after security training that incorporated game elements (DOI: 10.1080/07421222.2019.1705512). The abstract of this study does not clarify whether that behavioral change measured phishing email clicks separately.

While the effects of a one-time training session have been confirmed on a scale of days to weeks, evidence that those effects last beyond several months remains sparse.

A Busy Afternoon, a Single Click

Let us follow the gap between the laboratory and the field through the afternoon of a single person. An employee at a hospital receives a simulated phishing email during a busy afternoon rush.

The conditions under which this employee receives the email differ from those in the laboratory. Early laboratory studies on embedded training tested participants immediately after training (DOI: 10.1145/1299015.1299022). The next simulated email arrives weeks later for this employee.

The workload is also different. When matching hospital employees' survey responses with actual click records, the intention to follow rules had no clear relationship with the actual behavior of not clicking the link. The only factor that went along with the likelihood of clicking was the employee's workload (PMID 32012071). Answering "I will be careful" on a survey and not clicking during a busy afternoon were two different things.

The worries at the moment of clicking are also different. Looking at multi-year training records for about 70 employees at a US government research agency, people who clicked the link worried about what would happen if they did not—such as failing to respond on time. People who did not click worried about what would happen if they did—such as downloading malicious software (DOI: 10.14722/usec.2018.23016). The same email was read differently depending on a person's workload situation.

And when the link is clicked, a training screen appears. In a large-scale experiment comparing the wording styles of training messages, researchers could not determine whether the wording changed effectiveness, because employees did not read the training materials (DOI: 10.1109/msp.2013.106). Even if the screen appears, no opportunity to teach arises if it is not read.

These facts came from different studies. No study directly measured whether these factors overlapped for a single person to erase the effect in the field, among the evidence gathered in this article. What becomes visible when placing them side by side is one thing. The conditions measured by early laboratory studies and the conditions under which a field employee opens an email differed in timing, workload, worries at the moment of clicking, and whether the training screen was read.

What a Single Click-Through Rate Misses

Clicking a link and giving away a password are different steps. A 15-month study did not measure just one thing either. It measured the click-through rate of phishing emails, risky behaviors such as typing in a password, and emails that employees reported as suspicious (DOI: 10.1109/sp46214.2022.9833766). In a test at a Portuguese company, a large proportion of the people who clicked the link in a simulated email went so far as to hand over personal information in some form to the fake site (DOI: 10.1109/ecrime.2008.4696970).

The reporting rate is the proportion of people who received a simulated phishing email and reported that email as suspicious. If the click-through rate counts the people who were fooled, the reporting rate counts the people standing on the side that blocks them. In the 15-month study, the company added a report button to the email program. When they collected reports from employees, they could spot new phishing attacks faster, the amount of work the organization had to handle was manageable, and employees kept reporting consistently over a long period (DOI: 10.1109/sp46214.2022.9833766). If you look only at the click-through rate, this achievement is not captured in the numbers.

The click-through rate moves not only with the people, but also with the emails. In a first simulation drill conducted at a large Italian hospital with over 6,000 employees, 64% of employees did not open a generic email that could be sent to anyone, while only 38% did not open an email tailored to the hospital's circumstances. Employees who clicked the link were also clearly more numerous in the tailored email group (PMID 35321019). A study that sent various simulated attacks to over 31,000 participants built a model to predict in advance how many people would be fooled based solely on the email's characteristics. The goal is to gauge how plausible that email is before sending out the training (DOI: 10.1109/access.2022.3207272). If how the email was decorated moves the numbers this much, the difference in click-through rates between two training sessions may contain both changes in people and differences in the emails.

The tilt of the judgment also moves the numbers. Response bias is the tendency to judge emails overall as real or fake, separate from the ability to tell each individual email apart. Commonly used intuitive metrics mix this bias and accuracy together (PMID 30063406). In an online email classification task, knowledge about phishing went along with better at spotting phishing, but also with a bias to classify all emails as phishing (PMID 38729025). When clicks decrease, whether people got better at recognizing fakes or simply started clicking any email less is a different story.

There is also evidence in the opposite direction. In a 515-person test, training did not reduce the intention to click links in legitimate emails (DOI: 10.1145/1572532.1572536). At least in this test, the training did not make people suspicious of all emails.

The mere fact that the click-through rate went down does not separate whether people got better at spotting fakes, whether the emails were easy, or whether they started clicking all emails less. If you measure responses to legitimate emails alongside it like in the 515-person test, you can at least tell apart whether they simply started clicking all emails less.

The Same Person Clicks Again

Among 5,416 healthcare workers who received all twenty simulated emails, 975 people—17.9%—never clicked any, while 3,565 people (65.3%) clicked two or more times. The 772 people who clicked five or more times were separately classified as a high-risk group (PMID 30861069). Mandatory training was given to this high-risk group starting after the fifteenth campaign, but that training had no major impact on click rates, and the high-risk group kept clicking more until the end (PMID 30861069).

In other settings, too, those who were fooled before were fooled again. In a study that sent simulated phishing to 6,938 faculty and staff at George Mason University in the United States, people who had fallen for phishing before were more likely to fall for it this time as well. The effects of gender and age were unclear or weak (DOI: 10.1145/3461672).

That is why arguments arise to target the people who are fooled the most. In a simulation calculation run by plugging in the figures from the preceding studies, within that model's assumptions, the most vulnerable users accounted for a large share of the total phishing risk, and the net benefit of interventions targeting them was much greater. This calculation also estimated that the net benefit of interventions reducing response bias was greater than improving the ability to screen them out (PMID 29023908). The preceding research team, which built a model to gauge the plausibility of emails in advance, also concluded that training should be tailored to individual users rather than large groups (DOI: 10.1109/access.2022.3207272).

When you place the two pieces of evidence side by side, one blank space becomes visible. We were able to find out who clicks repeatedly. However, the mandatory training separately given to those people did not significantly change click rates at the healthcare institution mentioned earlier. At least in this healthcare institution, finding dangerous people and changing their behavior were not the same thing.

The Cost of Training That Deceives Employees

Simulated phishing training only works if it tricks employees. Phishing experiments that mimic real conditions violate widely accepted informed consent requirements and involve deception. The authors of one ethical analysis argue that it can still be done ethically if it minimizes risk, protects confidentiality and privacy, offers an opportunity to opt out before starting, and debriefs participants afterward (PMID 28812222).

Problems also arose in the field. At the Italian hospital mentioned earlier, the training did not proceed as planned due to issues raised within the hospital. The summary does not disclose what those issues were. Concluding that simulations are useful but have limitations, the researchers noted that messages containing false threats or promises can provoke backlash from employees and unions. They also noted that successful and ethical simulations require organization-wide coordination, precise timing, and the condition that employees remain unaware (PMID 35321019).

Simulated phishing training only works when employees are deceived without knowing it, and that deception can trigger internal organizational backlash. This cost does not show up in click-rate tables.

Why the Results Diverge

If we place both sets of evidence on the same scale, it looks like this:

What the study in the final row tested was not training, but a brief prompt that touches on the discomfort arising when beliefs and actions diverge right before a simulated phishing attempt. Among the employees who were randomly assigned and actually responded to the survey, 44% (14 out of 32) in the stimulated group and 65% (22 out of 34) in the control group clicked the link, while 53% (396 out of 753) of the remaining employees who did not respond to the survey did so. However, this difference was not statistically clear, and the researchers noted that these results do not prove a reliable behavioral effect (PMID 42224651).

Reviews that bring studies together point to the same place. In a meta-analysis pooling 48 field experiments conducted in actual organizations, the average rate of falling for phishing emails was 21%. Training was one of the factors altering that rate, alongside warning devices, the degree of email personalization, and already known deceptions. Success rates also appeared to vary by orders of magnitude depending on the type of fraud (DOI: 10.1109/ecrime47957.2019.9037502). Because training is not the sole factor moving the rate of falling for attacks, comparing studies with one another or measuring before and after training using different emails can cause the choice of email to become mixed into the training's performance. The authors noted that many results had limitations in control and sample size, which could explain unexpected and mutually contradictory outcomes (DOI: 10.1109/ecrime47957.2019.9037502).

Reading down the columns of the table and looking at who conducted the studies, at least five reasons emerge for why the results diverge. The first is what is being measured. Clicks, password entries, reports, discrimination scores on experimental tasks, and employees' self-perceived effectiveness all count different things. At the Italian company, perceived effectiveness and actual outcomes diverged. The second is the timeframe. Effects measured 10 days or 28 days later and results accumulated over 15 months do not answer the same question. The third is the comparison group. The comparison where embedded training showed an effect was against the practice of sending security notices, whereas the comparison where mindfulness training showed an effect was against rule-based training. The fourth is the shape of the training. In a meta-analysis pooling 19 studies and 23,146 participants that tested interventions against social engineering attacks, some interventions had massive effects while others had no effect at all. High-intensity interventions were more effective than low-intensity ones, and narrowly focused interventions were more effective than broad ones (DOI: 10.1108/ics-07-2019-0078). This means that under the single name of "training" or "education," things with entirely different effects can be lumped together. The aforementioned literature review of 42 papers also noted that training intensity, active learning, detailed feedback, and combining traditional cue-spotting methods with training that teaches attention-paying skills are the factors that determine effectiveness (DOI: 10.1016/j.cose.2023.103695). The fifth is who did the measuring. The two field trials where embedded training showed an effect were conducted by the researchers who created the method, whereas the test at the Italian company and the 15-month study—where no effects were seen—were conducted by other researchers.

The conclusion of the same literature review embraces both sides. While the improvement in users' resilience against phishing emails confirms that training is a crucial defense tool, current methods still leave trained individuals at risk (DOI: 10.1016/j.cose.2023.103695). Even in a simulation with 41 participants where both training methods helped safe behavior, most participants still fell for phishing after training. The researchers concluded that training alone falls short of making people behave securely (DOI: 10.3390/fi14040104).

What the Fake Doorbell Teaches

Here is where conventional wisdom is right. In workplaces that repeated simulated phishing, fewer people clicked, and raising awareness or letting people experience the risk reduced the number of employees who handed over their passwords. The method of teaching at the moment of the click also showed effects in a short-term trial by the researchers who created it.

Here is also where conventional wisdom falls short. In a study following more than 14,000 employees for 15 months, embedded training using the exact methods commonly used in the industry did not make employees stronger. The effect of a single training was usually confirmed in units of a few weeks, and there is little evidence that the effect lasts beyond several months. The drop in clicks at workplaces that repeated simulated emails for years is an observational record, which is a different question from whether a single training lasts a long time. Separate training assigned to repeat clickers did not significantly change the click rate at one healthcare institution. And the click rate alone does not show everything that has changed.

In the same 15-month study, what worked well were warnings attached to emails, and what lasted a long time were employees' reports (DOI: 10.1109/sp46214.2022.9833766). Training by ringing a fake doorbell is one of several devices protecting the door, and its effectiveness looks different depending on what kind of training is measured with what and when.

Which moments of a person phishing targets, and who falls for which emails, are covered in "Phishing and Social Engineering — Ringing the Doorbell Instead of Breaking the Lock". What two-step verification blocks and what it fails to block when a person eventually hands over their password is continued in "The Limits of Two-Step Verification — Which Visitors Can Still Open the Second Door?".

Source Papers

• "The Phishing Email Suspicion Test (PEST) a lab-based task for evaluating the cognitive mechanisms of phishing detection." (2021) — PMID 33078362 — High volume and designed to deceive make purely technical solutions limited, with human judgment as the final line of defense.
• "Assessment of Employee Susceptibility to Phishing Attacks at US Health Care Institutions." (2019) — PMID 30848810 — 6 US healthcare institutions, 2011–2018, 422,062 out of 2,971,945 phishing simulation emails (14.2%) clicked. Institutional median ranges from 7.4% to 30.7%. Repeated campaigns associated with decreased click odds, suggesting potential benefits of simulated training. Retrospective and convenience sample.
• "Evaluation of a mandatory phishing training program for high-risk employees at a US healthcare system." (2019) — PMID 30861069 — 5,416 employees and 20 rounds. 975 employees (17.9%) with 0 clicks, 3,565 (65.3%) with 2 or more, and 772 high-risk individuals with 5 or more. Click rates decreased across all groups, with mandatory training for high-risk individuals showing no major impact after 15 rounds.
• "Informing, simulating experience, or both: A field experiment on phishing risks." (2019) — PMID 31851688 — Field experiment with over 10,000 Dutch government ministry employees. Both information provision and simulated experience significantly reduced password submission rates, and combining them did not further increase the effect.
• "Protecting people from phishing" (2007) — DOI: 10.1145/1240624.1240760 — Laboratory experiment. Embedded training is more effective than security notice practices.
• "Getting users to pay attention to anti-phishing education" (2007) — DOI: 10.1145/1299015.1299022 — Previous research tested immediately after training. Embedded training is superior to sent-email training in learning, retention, and transfer.
• "Lessons from a real world evaluation of anti-phishing training" (2008) — DOI: 10.1109/ecrime.2008.4696970 — Field test at a Portuguese company. A large proportion of those who clicked entered information. Trained individuals were less deceived a week later. Effects were observed to spread even to colleagues who had not received direct training.
• "School of phish" (2009) — DOI: 10.1145/1572532.1572536 — Real-world test with 515 people. Knowledge retention remained after 28 days, a second training message reduced information provision, and the intent to click real links did not decrease.
• "A real world study on employees' susceptibility to phishing attacks" (2020) — DOI: 10.1145/3407023.3409179 — Real phishing test with 191 employees at an Italian company. Embedded training was perceived as effective but did not reduce vulnerability.
• "Going Spear Phishing: Exploring Embedded Training and Awareness" (2013) — DOI: 10.1109/msp.2013.106 — Large-scale experiment, 4 training presentation methods showed no significant difference in subsequent clicks. Many clicked everything or nothing regardless of training status. Employees did not read the training materials, making it impossible to judge effectiveness.
• "Phishing in Organizations: Findings from a Large-Scale and Long-Term Study" (2022) — DOI: 10.1109/sp46214.2022.9833766 — Over 14,000 people and 15 months, measuring clicks, risk behaviors, and reporting. Email warnings work well. Industry-style embedded training cannot be made stronger and may have side effects. Reporting group detection is fast, manageable in burden, and shows long-term participation.
• "Exploring the evidence for email phishing training: A scoping review" (2023) — DOI: 10.1016/j.cose.2023.103695 — 42 papers. Short-term effects are well-documented, but evidence for sustained behavioral change is limited. Effective elements are a combination of intensity, active learning, detailed feedback, and attentional skills paired with cue-based knowledge. Training is an important defense, but current methods still leave risks.
• "Suspicion, Cognition, and Automaticity Model of Phishing Susceptibility" (2016) — DOI: 10.1177/0093650215627483 — Evidence that training effects fade within a short period as people fall back into email habit patterns.
• "Embedding Training Within Warnings Improves Skills of Identifying Phishing Webpages." (2019) — PMID 30526089 — Online experiment. Warnings incorporating education provided better protection in the subsequent stage without a warning and at the one-week follow-up stage compared to conventional warnings.
• "Is the key to phishing training persistence?: Developing a novel persistent intervention." (2022) — PMID 35157483 — Results from traditional one-time interventions are limited. In an experiment with college-age adults, a persistent classification helper increased detection, with the greatest effect when embedded directly within tasks.
• "Why Employees (Still) Click on Phishing Links: Investigation in Hospitals." (2020) — PMID 32012071 — Survey of 397 hospital employees compared with actual clicks. Compliance intention has no significant relationship with compliance behavior, and only workload is positively correlated with click likelihood.
• "User Context: An Explanatory Variable in Phishing Susceptibility" (2018) — DOI: 10.14722/usec.2018.23016 — About 70 US government research agency employees, 4.5 years of embedded training materials. Those who clicked worried about failing to respond, while those who did not clicked worried about malware. Work context dictates cue interpretation.
• "Phishing simulation exercise in a large hospital: A case study." (2022) — PMID 35321019 — Italian hospital with over 6,000 employees. Unopened general mail 64% vs tailored mail 38%, with more clicks on the tailored side. Organizational issues prevented the plan from proceeding as scheduled. False threats and promises may provoke backlash from employees and unions. Coordination, timing, and staff unawareness are required.
• "Avoiding the Hook: Influential Factors of Phishing Awareness Training on Click-Rates and a Data-Driven Approach to Predict Email Difficulty Perception" (2022) — DOI: 10.1109/access.2022.3207272 — Over 31,000 people and 144 types of mock attacks. A model predicting deceived individuals based on email characteristics and estimating prior difficulty. Concludes that training should focus on individual users rather than large groups.
• "Signal Detection Theory (SDT) Is Effective for Modeling User Behavior Toward Phishing and Spear-Phishing Attacks." (2018) — PMID 30063406 — Intuitive metrics confound response bias and accuracy. Signal detection metrics can be used to evaluate and monitor training efficacy.
• "The roles of phishing knowledge, cue utilization, and decision styles in phishing email detection." (2024) — PMID 38729025 — 145-person online classification task. Phishing knowledge is associated with better detection alongside a bias to classify all emails as phishing.
• "Experimental Investigation of Technical and Human Factors Related to Phishing Susceptibility" (2021) — DOI: 10.1145/3461672 — George Mason University faculty and staff of 6,938, 3-week campaign. Those previously phished are more easily deceived again, while gender and age effects are insignificant or weak.
• "Setting Priorities in Behavioral Interventions: An Application to Reducing Phishing Risk." (2018) — PMID 29023908 — Monte Carlo simulation using parameters from existing studies. The most vulnerable users account for a large share of the risk, and interventions targeting them yield high net returns. Interventions that reduce response bias rather than increasing sensitivity yield higher net returns.
• "Ethics and Phishing Experiments." (2018) — PMID 28812222 — Argues that real-world condition phishing experiments violate informed consent requirements and involve deception, but can be conducted ethically if risks are minimized, confidentiality is protected, an opt-out opportunity is provided, and debriefing is performed.
• "Training to Mitigate Phishing Attacks Using Mindfulness Techniques" (2017) — DOI: 10.1080/07421222.2017.1334499 — Field study of 355 US university students, mock attack 10 days later. Mindfulness training helped avoid attacks better than rule-based training.
• "Using Design-Science Based Gamification to Improve Organizational Security Training and Compliance" (2020) — DOI: 10.1080/07421222.2019.1705512 — 420 people and 6-month field study. Gamified security training showed statistically significant positive behavioral changes.
• "A meta-analysis of field experiments on phishing susceptibility" (2019) — DOI: 10.1109/ecrime47957.2019.9037502 — Meta-analysis of 48 field experiments. Average deception rate 21%. Out of 140 tests of recipient variables, 116 were mostly insignificant. Warning devices, personalization, training, and known methods alter rates, with orders-of-magnitude differences depending on the scam type. Controls and sample limitations can explain unexpected or contradictory results.
• "How effective are social engineering interventions? A meta-analysis" (2020) — DOI: 10.1108/ics-07-2019-0078 — 19 experimental design studies, 37 effect sizes, 23,146 participants. Highly effective interventions and ineffective interventions coexist. High intensity and a narrow focus are more effective.
• "Cognitive Dissonance-Based Priming Intervention: Randomized Encouragement With in-the-Wild Phishing Simulation Attack in Health Care." (2026) — PMID 42224651 — 819 employees at a large Norwegian hospital. Click rate control 65% (22/34), stimulus 44% (14/32), non-response 53% (396/753), P=.22, which is not significant. Does not prove reliable behavioral effects.
• "Evaluation of Contextual and Game-Based Training for Phishing Detection" (2022) — DOI: 10.3390/fi14040104 — 41-person simulation experiment. Both trainings helped safe behaviors, and contextual micro-training was better, but most were still deceived after training → training alone is insufficient.