You can’t just wish away the capacity for violence. Nor can you deprive aggressors of the means of violence except by way of good people having the ability and willingness to use defensive force. You, personally, don’t have the right or the ability to decide which of the other several billion people on the planet possess the means to use force, including deadly force. And feeling anxiety about that doesn’t make it untrue.
But by far the worst thing you could possibly advocate or hope for is a monopoly on the ability to use force. If the capacity for violence is all consolidated in one place, the most evil people on the planet will do absolutely anything to get control over that monopoly. And when (not if) they do, you will have just given evil a massive advantage and head start.
You can want an all-powerful Tooth Fairy to make sure justice always prevails. But wanting it won’t make it exist. And a quick look at history shows that attempting to create an all-powerful, irresistible “protector” force always results in making an all-powerful, irresistible aggressor force.
To put it in more plain terms, if you’re a fan of violence, suffering, oppression, and mass murder, then advocating that only “government” agents be armed is a great idea. Otherwise, it’s about the stupidest thing you could possibly condone.