Responsible Rulers

I have often heard people charge libertarian anarchists with being irresponsible for wishing to get rid of the current system of government and replace it with genuine self-governance. It’s as if — however difficult it may be to believe — these critics actually believe that rulers in the current setup are responsible.