Who’s the Aggressor? Who’s the Victim?

When a libertarian says that the most basic individual right is the right not to be aggressed against, a clever interlocutor may accuse the libertarian of begging the question, of stuffing the rabbit into the hat. The trick, the critic will say, is in the word aggress: libertarians allegedly rig the game by restricting the category of aggression to only the actions they disapprove of, thereby institutionalizing many corrupt activities.