PhishTriage

The finding

On a small test set of 33 emails, 18 synthetic and 15 real ones from my own inboxes, the tool gets 28 right at the default threshold with no false alarms. The five it misses are the interesting part: four were sent from compromised student accounts inside the university’s own Microsoft 365 tenant, and one is a fake-interview scam sent through a legitimate mail provider with SPF and DKIM passing. In every case the infrastructure is clean and only the text is wrong. The tool caps text at 15 points on purpose, so it cannot convict on wording alone, and the misses are documented in a file the regression test enforces.

Context

During my summer 2026 co-op I triaged 294 user-reported emails by hand, and the first few minutes on every one were the same checks: who sent it, did SPF, DKIM and DMARC vouch for them, does the domain that authenticated match the one displayed, where did it physically come from, and what does it want me to click or open. I wanted that checklist written down as rules, with every point on the score traceable to one of them.

How it was built. I chose the problem, the explainable rules-over-machine-learning design, and a command-line tool over a web interface. Claude Code generated most of the Python from a specification it drafted at my direction. My part after that was reviewing the output, running the tests, supplying and labeling the real email samples, and deciding how to handle calibration and the known misses. I am still working through the code so I can explain every subsystem, and caching, rate limiting and attachment magic-byte checks are the parts I know least well today.

What was built

A command-line analyzer for .eml files:

• Parsing that runs two passes and downgrades problems to warnings, so a malformed message produces a report instead of a stack trace.
• Authentication and alignment. SPF, DKIM and DMARC results are read from the Authentication-Results, ARC-Authentication-Results and Received-SPF headers, then checked for alignment: Return-Path against From, DKIM d= against From, Reply-To on a different domain, and display names that name a brand or embed another address.
• Hop analysis. The Received: chain is reversed to origin-first, the boundary hop where the recipient’s provider accepted the message from outside is found, and the external origin IP, private IPs presented as external, and timestamps out of order are all flagged.
• Indicators. URLs from text and HTML with visible-text mismatches, lookalike and typosquat domains against a brand list, shorteners, raw IPs, punycode and deep subdomains. Attachments are hashed and type-sniffed for double extensions, executables, macro-enabled Office files and container formats. Nothing is opened, executed or fetched.
• Enrichment, optional and API-only: URLhaus, VirusTotal and RDAP domain age, cached on disk for a day and rate-limited to the free tiers.
• Scoring from data. Every signal maps to a rule in a YAML file with an id, weight, category, condition and a one-sentence rationale. The scorer sums what fired, caps at 100, and prints a verdict with the evidence behind every point. No machine learning, by design: each point traces to a rule someone can read and argue with.
• A report you would paste into a ticket, in the terminal, as JSON or as Markdown, with indicators defanged by default. A batch mode prints a confusion matrix for a folder.

There are 163 tests at 93% coverage, a CI workflow, a sanitizer for turning real inbox exports into publishable samples, and a DECISIONS.md that records why each weight is what it is.

Evidence

The case the tool was built to make visible: SPF and DKIM both pass, and it is still a phish, because the domain that authenticated is not the domain in From.

Three of the rules, verbatim:

- id: dmarc_fail
  description: DMARC check failed
  category: authentication
  weight: 25
  rationale: The domain owner published a policy and this message failed it, which is the strongest single header-based spoofing signal available.
  when: {field: auth.dmarc.status, in: [fail, permerror]}

- id: display_name_spoof
  description: Display name impersonates a brand or embeds a different address
  category: sender
  weight: 20
  rationale: Mobile clients show only the display name, so "Microsoft Support <random@freemail>" looks legitimate to most recipients.
  when: {field: auth.display_name_spoof, is_true: true}

- id: lure_keywords
  description: Urgency, credential, financial, threat or reward language
  category: content
  weight: 5
  weight_per: keyword_category
  max_weight: 15
  rationale: Social-engineering lures lean on pressure and fear; the language alone is weak evidence but it compounds with everything else.
  when: {keywords: true}

Results on the whole corpus at threshold 50:

Set	Phish caught	Benign cleared	Missed	False alarms
Synthetic (12 + 6)	12/12	6/6	0	0
Real inbox mail (6 + 9)	1/6	9/9	5	0
Whole corpus (18 + 15)	13/18	15/15	5	0

The misses live in samples/known-misses.txt. The regression test requires each one to stay below 50 and above a documented floor, so the list cannot go stale and a fix cannot go unnoticed. The batch command does not consult it: the confusion matrix reports the misses honestly.

What it means

• Authenticated is not aligned. Passing SPF and DKIM only says some domain vouched for the message. Whether it was the right domain is a separate question, and it is the one that catches most lookalike and relay-based phish.
• Clean infrastructure is the hard case. Mail from a compromised account inside your own tenant passes every header check because it should. Detecting it is a content and behaviour problem, and a tool that pretends otherwise by inflating keyword weights would start convicting newsletters.
• Real mail broke assumptions the synthetic set could not. Microsoft 365 and Gmail never name the recipient’s domain in their hop chains, so boundary detection had to work in provider families. Tenants without custom DKIM sign with onmicrosoft.com, so alignment rules now only score when DMARC itself failed. And storage.googleapis.com is Google, not a Google lookalike.
• A shared host’s reputation describes its other tenants. With live threat intel, URLhaus and VirusTotal scored a real phish on storage.googleapis.com as a whole, not on the link in the email, and pushed it to a raw 120. Reputation rules now skip shared hosting and still count a hit on the exact URL. The same flaw still applies to URL shorteners and marketing click-trackers, and that fix is next; Right verdict, wrong reason covers it.
• Rules as data pays off at review time. Every weight change is a text diff, the corpus re-scores in a test run, and the reasoning is in the file next to the number.