This paper presents a search for decays of the Higgs boson with a mass of 125 GeV into a pair of new pseudoscalar particles, $H\to aa$, where one $a$-boson decays into a $b$-quark pair and the other into a muon pair. The search uses $139{\mathrm{fb}}^{-1}$ of proton-proton collision data at a center-of-mass energy of $\sqrt{s}=13\mathrm{TeV}$ recorded between 2015 and 2018 by the ATLAS experiment at the LHC. A narrow dimuon resonance is searched for in the invariant mass spectrum between 16 GeV and 62 GeV. The largest excess of events above the Standard Model backgrounds is observed at a dimuon invariant mass of 52 GeV and corresponds to a local (global) significance of $3.3\sigma$ ($1.7\sigma$). Upper limits at 95% confidence level are placed on the branching ratio of the Higgs boson to the $bb\mu \mu$ final state, $\mathcal{B}(H\to aa\to bb\mu \mu )$, and are in the range $0.2–4.0\times {10}^{-4}$, depending on the signal mass hypothesis.

• Received 4 October 2021
• Accepted 22 November 2021

DOI:https://doi.org/10.1103/PhysRevD.105.012006

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Funded by SCOAP³.

© 2022 CERN, for the ATLAS Collaboration