Max asks:

Can anyone explain by what legal authority a U.S. Marshall can order a journalist to erase a tape recording of statements made in a public place? If only someone with legal expertise had been on the scene.

Eugene Volokh considered the same event and says:

If this report is accurate, then I don’t see any legal justification for the marshal’s demand, or the marshal’s seizing the tape recorder (which therefore sounds like a Fourth Amendment violation to me). To my knowledge, there’s no law — it would presumably have to be a Mississippi law — prohibiting tape recording of public events, even ones on private property.

This practice seems to be common in other contexts. For instance, theater and concerts come immediately to mind. How is Scalia’s practice different from, for example, Bob Dylan’s?
Note, I in no way support Scalia’s practice. He is a civil servant and as such should be 100% transparent in all work and public activities.
On the other hand, I think musicians are being foolish when they prohibit recording. Of course, in many cases (not Dylan) it would take only a few concert recordings to circulate to expose the complete lack of creativity they bring to the stage.