If diversity means anything, it means that different groups differ at least a little in what they want and in what talents they bring, as well as in how they are treated by others.
Privilege is real - some groups have unearned advantages over others, which does increase the chance that people like that will be disproportionately represented among those in power.
But it is also the case that different groups want different things, or more exactly, the proportions of people who want some outcome will not be identical in each group.
It is also true, but harder to pin down, that different groups bring a different mix of skills and talents to life. This difference has to have some effect on the proportion of people from each group who end up doing one thing or another.
It is certainly the case that the group of people in positions of power are disproportionately from privileged groups. But privilege is only part of the reason. To say otherwise, to say that all groups have an exactly equal desire and exactly equal ability to achieve all outcomes is simply false. Worse, it is condescending and imperialist to assume that all groups - all cultures and subcultures - desire exactly the same things equally.
In the discourse about privilege, to assume that the privileged are disproportionately powerful solely because we are privileged is itself an act of unwarranted privilege.