Executing Shell Commands in Emacs

Interacting with external commands in Emacs ranges from simple one-off commands to launching interactive sessions using comint, the Emacs command interpreter library. In earlier articles I covered shell integration and terminal emulation in Emacs, but this time I want to talk about the other use case: executing shell commands and capturing their output; how to use them in elisp code; how to make them work with unicode and different coding systems; and a few unique ways of using the commands.

Executing Shell Commands

There are several shell commands in Emacs – not to be confused with M-x shell, the Emacs inferior shell – and they are tightly integrated with Emacs’s buffer system.

The Basics

The simplest way to invoke something is to run the command shell-command, bound to the handy shortcut M-!. Emacs will try to display the output of the command in the echo area if it is not too large; otherwise, it will be sent to the *Shell Command Output* buffer.

Emacs will display the exit code of the terminating command in the mode line if it is non-zero. Any output sent to standard error will be interspersed as though it came from standard out; if you want it sent to its own buffer you must set shell-command-default-error-buffer to the name of a buffer you want it sent to.

By default Emacs will execute the command synchronously; that is, it will block Emacs until the command has run its course. If you want Emacs to invoke it asynchronously, you must append an ambersand (&) or use the asynchronous version, async-shell-command bound to M-& (note: it was introduced in Emacs 23.2.) The asynchronous version will use the buffer *Async Shell Command* for its output.

If you prefix the command with the universal argument, C-u, Emacs will insert the output of the shell command into the current buffer at point.

This feature is a god send, for mostly obvious reasons. But it does let you do one cool thing I haven’t seen other people talk about before: you can use it in conjunction with recursive minibuffers.

It’s Recursion All The Way Down

So, a brief digression: if you set enable-recursive-minibuffers to t you can invoke interactive commands from within an “active” minibuffer command and use its output in the parent minibuffer command. Without the flag set to true, this will not work. Emacs will throw an error saying you can’t use a minibuffer command inside another one.

Confused? Well, consider the following use case: you have a text file with a bunch of file paths and you want to replace all matches of the current directory the file you are editing is in.

So if you’re editing /tmp/very/long/and/terse/path/to/foobar.txt and you want to replace all occurrences of that path in the file with something else, you’d have to either type it in or kill it from some place else and then yank it. Not with shell commands and recursive minibuffers, you won’t!

Mark the contents of the file you want to edit and invoke query-replace (M-%) and now type C-u M-! pwd RET and hey presto. Emacs will have inserted it into your minibuffer instead of your current buffer, and it will take you back to where you were before — preparing a query replace.

Note:Because shell commands end with a newline you’ll have to delete the newline character from the output by typing C-e, to move to the end, and C-d to delete it.

Shell Commands on Regions

Unlike the “normal” shell commands I explained before, this one works on regions of text you mark in a buffer. The text is sent to the command – which must read from stdin – and the output, if any, is captured. To use it, you can invoke either shell-command-on-region or its shortcut, M-|. The marked text is sent, verbatim, to the shell command and its output. If you want Emacs to replace the region with the output returned from the command, you should prefix the command with the universal argument, C-u.

If shell-command-default-error-buffer is set, as with the aforementioned commands, then an error buffer, and any text sent to standard error, is displayed there instead.

It goes without saying that this is an incredibly useful feature. Data scrubbing and text manipulation is made so much easier as you have every commandline tool at your finger tips; this is especially useful for those of you with commandline-fu but are not yet familiar with all of Emacs’s advanced text editing features. Furthermore, it is perfectly possible for you to string together multiple, piped commands and use that as a shell command. You are not limited to just one command.

Let’s Go Meta

You can use the shell region command to invoke shell code (or python code, or..) by passing something like sh, bash, or python as a command. You may have to adjust shell-command-switch if you use it on esoteric interpreters or shells (though the default will work with sh, bash and python.)

Coding Systems and Unicode

By default Emacs will use your buffer’s coding system to determine how to send and receive characters from the shell. Whether this will work for you or not will depend on your platform, and so on. But assuming that does work you may find that either Emacs or the shell screw up the encoding. Thankfully, it is easy to fix.

The easiest thing is to set the buffer’s file coding system. To change the coding system you must type C-x RET f. To enumerate the list of coding systems, type M-x list-coding-systems.

The other option, if you want to use a different coding system from the one in your buffer – or if you simply do not want to change from the usual dos, mac or unix format – is to use the very handy universal-coding-system-argument command, bound to C-x RET c. Basically, it asks for a coding system – such as utf-8 – and a command to invoke with this coding system; in this case it’s M-! or M-| or, indeed, any command you choose.

To summarize: if you get garbled output or missing unicode characters, make sure your coding system is set properly!

Invoking Shell Commands from Elisp

You can use the same commands you used in interactive mode in your elisp code, with the only catch that you have to fill out a few more parameters.

Here’s a commented example I wrote that marks the entire buffer and feeds it to tidy, a useful utility for cleaning up and formatting HTML and XML:

(defun tidy-html ()
  "Tidies the HTML content in the buffer using `tidy'"
  (interactive)
  (shell-command-on-region
   ;; beginning and end of buffer
   (point-min)
   (point-max)
   ;; command and parameters
   "tidy -i -w 120 -q"
   ;; output buffer
   (current-buffer)
   ;; replace?
   t
   ;; name of the error buffer
   "*Tidy Error Buffer*"
   ;; show error buffer?
   t))

This function will run tidy with a few settings on the entire buffer, replace the output, and if there are errors it will display them in a separate, designated buffer.

Conclusion

There’s lots to be said about shell commands, and I think I’ve managed to cover most of it. The commands are surprisingly versatile and in true Emacs fashion has a wide variety of applications beyond the more obvious ones. Computer games have Emergent Gameplay; we have Emergent Emacs. I also think – even though I have barely scratched the surface – this shows off Emacs’s awesome unicode system. It’s so incredibly advanced and flexible. I hope to one day do an in-depth article on all the subtleties and nuances of Emacs’s awesome unicode system.