Wenlong Hou

As Large Language Models (LLMs) are increasingly deployed in healthcare worldwide, robustness against adversarial jailbreaking becomes essential for maintaining medical ethics compliance, calling for comprehensive adversarial benchmarks. However, existing benchmarks remain predominantly English-centric and limited to single-turn attacks. To address these gaps, we introduce JMedEthicBench, to our knowledge the first multi-turn adversarial benchmark for evaluating medical ethics alignment of LLMs in Japanese via static-replay pressure test. Grounded in 67 guidelines from the Japan Medical Association, our benchmark comprises 50,843 adversarial conversations generated using seven automatically discovered jailbreak strategies, which we further show cluster into two adversarial families. Through a dual-LLM scoring protocol validated on 653 human-annotated responses (Spearman $\rho = 0.775$ vs. the dual-judge average), we evaluate 22 models and find that many commercial models maintain strong resistance to attacks while medical-specialized models exhibit increased vulnerability. Furthermore, safety scores decline significantly across conversation turns (median: 9.5 to 5.0, $p < 0.001$), with $82.3\%$ of all conversations showing at least one step of safety decline and $50.1\%$ of medical-model conversations following a strictly monotonic decline. Cross-lingual evaluation reveals that medical model vulnerabilities persist across Japanese and English, indicating inherent alignment limitations rather than language-specific factors. Our findings suggest that domain-specific fine-tuning may inadvertently weaken safety alignment when it is not explicitly preserved, and that multi-turn jailbreak attacks represent a distinct threat surface requiring dedicated defense strategies.