When black-clad anti-fascist protesters broke through police barricades Sunday afternoon and swarmed a peaceful rally in Berkeley, California, law enforcement stood aside and let them. City police Chief Andrew Greenwood explained the decision with a rhetorical question: “Does it make sense,” he asked, “to get into a major use of force over a grassy area?”


With all due deference to police expertise, things are not that simple. Violent protesters who cross barriers and disrupt peaceful protest are deeply threatening to freedom of speech.


The “antifa” didn’t just block the speech of the handful of far-right protesters who continued on after their planned demonstration was canceled. The antifa also disrupted the 2,000-person anti-racism rally that was taking place to counter the far right. By breaching the peace, the antifa damped the free speech of the peaceful anti-racists.


Imagine that 100 neo-Nazis, instead of 100 anti-fascists, had crossed barricades in defiance of police orders. The public would have been justifiably outraged. And a police chief who argued that it was wiser to withdraw his officers rather than provoke confrontation would be under significant pressure to recant or even resign.


To be clear, my goal is not to criticize the police. Their thankless and crucial job of keeping the peace is the necessary condition for free speech. Avoiding violence should be a high priority, sometimes the highest. Contextual judgment is crucial.


The Berkeley police were on the ground and had to make a real-time judgment. I wasn’t there, and am relying on news reports and Greenwood’s words.


Rather, the Berkeley events raise the fundamental question of when police should intervene, by force if necessary, to protect peaceful speech from violent disruption.


Often, the situation is more clear-cut than it was in Berkeley, because violent protesters are trying to attack peaceful ones, with two distinct sides at odds. Under those conditions, the police are supposed to protect the peaceful speakers, not let the violent protesters shut them down.


If the police fail to protect peaceful protesters, then they’re allowing a version what is known as a “heckler’s veto”: Someone violating norms of civility is blocking the exercise of free speech by someone who is following the rules.


To the extent that the antifa were going after the handful of far-rightists on the streets in Berkeley, that’s exactly what was happening.


But because the far-right demonstration had been canceled for security reasons, the major event that was taking place was a peaceful anti-racism protest.


The antifa didn’t attack the peaceful anti-racists. But they affected their free speech rights nonetheless. They hijacked the peaceful protest by injecting violence into it.


Proof of this hijacking is that the national media that covered the story led with the antifa action and the police response, not the peaceful protest. Count that as a win for the extremists who broke the rules by defying police, not the peaceful protesters who followed the law.


The point is that while avoiding violence is a necessary goal, it is also a competing and important goal for the police to protect peaceful speech from violent threats. In the name of avoiding violence, the police must not cede public space to violent protesters - regardless of whether they are far left or far right.


That’s why it is worrisome to hear police say that it wasn’t worth fighting antifa over “a grassy area.” What was at stake wasn’t just a patch of grass, but the public space where peaceful speech was taking place. Protecting the First Amendment has to be a key police objective.


Consider that when a lone speaker is being met with large counterprotests, it’s always easier, cheaper and safer for police to remove the speaker than it is to control the counterprotesters. That’s the law of numbers.


Indeed, when Berkeley police led away a few far-right protesters for their own safety, they were just following this resource imperative.


Yet the effect of removing the isolated, vulnerable speaker faced with counterprotest is to silence the few while allowing the counterprotesters to prevail. That’s a violation of our free-speech ideals, which value everyone’s words equally, regardless of how many people are gathered to speak at a given location in a given moment.


When it comes to making moral judgments about the content of speech, there should be no false equivalence between racists and anti-racists. Racism must be condemned, and anti-racism applauded.


When it comes to the police enforcing free-speech rights, however, all speech must be treated identically. The state must be neutral with respect to the speakers’ viewpoints, and blind to their content.


The same is true for violent protest that threatens free speech. Government neutrality must be maintained so as to protect all peaceful speech equally.


The police must and should keep the peace. But they must simultaneously protect peaceful free speech - even if that requires the use of force.


Noah Feldman is a Bloomberg View columnist. He is a professor of constitutional and international law at Harvard University and was a clerk to U.S. Supreme Court Justice David Souter. His seven books include “The Three Lives of James Madison: Genius, Partisan, President” and “Cool War: The Future of Global Competition.”