There very much is an offshoot of extreme libertarianism that promotes anarchy as an ultimate political solution. Perhaps you're not familiar with that group. More prevalent is the libertarian minarchist.
I'd pose the same question about minarchism to you that I asked dross.
Sports are an example of minarchism. The referee doesn't play the game, he's just there to call fouls and enforce the rules. He doesn't make up rules as the game goes, he doesn't get to take a shot, he's just an observer. The only time the official becomes part of the game is during a dispute between players.
Why can't we operate government in the same fashion?