This is a reasonable perception within the Empire, with not a single flaw, and the same is true within the Alliance.
But what comes next varies between the Alliance and other Star Sectors.
For most residents in the world, I am a human being, an Imperial, the Governor is either an appointee or an enfeoffed ruler by the Imperial Faction, so he has the right to rule over me. And if the Empire replaces this Governor one day, then the new one also has the right to rule over me; if the Empire declares this Governor a traitor one day, I would even angrily take up arms to execute him myself.
The Alliance, however, is slightly different.
The legitimacy of the Alliance's rule principally emanates from direct appointments by the Empire, which is supremely important.