The Clifford hierarchy $\mathcal{C}_1 \subset \mathcal{C}_2 \subset \cdots$ was introduced by Gottesman and Chuang (Nature 402, 1999) to characterize gates that admit fault-tolerant implementation by gate teleportation. Yet, despite its rich mathematical structure and the attention it has received in recent years, little is known about $\mathcal{C}_k$ for $k > 3$. Most progress has focused on identifying structural properties of restrictions of the hierarchy, such as diagonal gates and gates on systems of small dimension $d$ or with few qudits.
The generalized semi-Clifford conjecture, proposed by Zeng et al. (Phys. Rev. A 77, 2008), states that every gate in $\cup_k \mathcal{C}_k$ is, up to multiplication by Cliffords, the product of a permutation and a diagonal matrix. Beigi and Shor proved the case $d=2$, $k=3$ (Quantum Inf. Comput. 10, 2010) and Pllaha et al. found an alternative proof by exploiting fixed points of the conjugation map induced by (a Clifford correction of) $U \in \mathcal{C}_3$ on the span of maximal stabilizer subgroups (Quantum 4, 2020). By extending their fixed-point arguments to the group $\Gamma_1(U)$ generated by $U \mathcal{P} U^\dagger$ and beyond, we prove the conjecture for $k \leq 4$ and any prime dimension $d$.
Our proof centers on conjugation groups $\Gamma_1(U), \Gamma_2(U), \ldots$ of $U \in \mathcal{C}_k$, which we expect to be a useful tool in the study of the Clifford hierarchy more generally. We also show a natural sufficient condition on such groups for gates in higher levels to be generalized semi-Clifford.