Human-Centered Security

The episode delivers solid empirical findings (2% average reduction, 20% reduction for interactive training users, 1-30% failure rate variance by lure type) with thoughtful measurement discussion. However, significant portions are consumed by banter, background-setting, and conversational padding. The core insights - that standard phishing training has minimal real-world effect because users don't engage with it, and that lure type matters more than training modality - are valuable but could be extracted more densely.

“the majority spent less than like 10 seconds or 30 seconds on it”

“there was about a 2% difference in average failure rate than folks who didn't get training...is all of this effort, all of this energy, all of this money worth a 2% reduction”

The work is original in execution (rigorous randomized controlled trial with real enterprise data, pre/post measurement of engagement), but the core finding - that security training has limited efficacy - is not new. The paper cites Lane et al. reaching similar conclusions. The framing around measurement rigor and the specific finding about engagement rates are novel, but the fundamental insight (training doesn't work well) is somewhat established in the literature.

“we were able to embed some metrics into the training that allowed us to see how long people were spending on the training”

“Lane et. al., was one of the ones that came before our paper. And they found, you know, very similar results, which is that phishing training has little to limited effect”

Ariana Mirian is a strong guest: PhD researcher from UC San Diego with direct involvement in enterprise security measurement research, currently a Senior Security Researcher at Censys. She has credibility as a practitioner-researcher with real institutional collaboration (UCSD Health, UCSD IT). She brings domain expertise and has published peer-reviewed work. However, she is primarily an academic researcher rather than an operator who has built or run security operations at massive scale in a commercial context.

“I currently work as a senior security researcher over at Censys, which is like the internet map of all devices and things on the world”

“we were very fortunate...to have a very close-knit relationship with like both the UCSD IT folks, and also the folks over at UCSD Health”

The episode provides concrete metrics: 2% average failure rate difference, ~19-20% reduction for interactive training engagers, 1-30% variance by lure type, majority spending under 10-30 seconds on training, 8 months of data, ~20,000 employees, specific training modalities (static generic, static contextual, interactive variants), and named lure examples (vacation policy, traffic violation, password reset). However, exact statistical significance thresholds, confidence intervals, and the specific phishing lures deployed are not all detailed in the conversation.

“there was about a 2% difference in average failure rate”

“interactive training, right, can reduce the likelihood of clicking on subsequent phishing emails by something like 20%...the actual number is 19”

Heidi asks relevant follow-up questions (research questions, dataset composition, why reporting wasn't tracked, recommendations) and shows genuine curiosity. However, much conversation is filler (background stories, tangents about moms, TikTok dancing, parking tickets, the Friday vibe, 'Phish Hoover' riffing) that delays substantive discussion. There are few probing disagreements or sharp pushes on findings; the host is warm but doesn't deeply challenge claims or drill into limitations of the measurement approach.

“One thing I wanted to ask you about...you, one of the things that you did not track was how many folks reported the phishing email, right?”

“if we pooled all of the money that we've spent/wasted on, on these, these trainings, could we have just built a better system?”