How to Execute Shell Commands Using Python

Python is a go-to for shell scripting and task automation. Google even uses it in their Automation courses. System admins reach for it too, since it can fire off shell commands with nothing but its default libraries. In this tutorial you’ll do exactly that: run Linux shell commands from Python, with a Raspberry Pi as the host computer.

Things you need:

• Computer
• Basic knowledge in using CLI (Command-Line Interface)

If you’ve never touched the CLI, or you just want a quick refresher, go through this tutorial first.

Introduction

Automation is everywhere right now. Home management, data extraction, those talking boxes in the kitchen, even self-driving cars. All of it runs on automation. Makes sense, since we’re living in the fourth industrial revolution. Heard of it? That’s the part where humans and machines try to work as one, and it’s happening now.

A Raspberry Pi opens a lot of doors here. With its GPIO, you can wire the credit-card-sized board into almost anything that sends or receives digital data. And since it’s a fully-fledged Linux computer, you can automate your computer tasks too. Python makes it way easier. There are two ways to run Linux commands from Python: the os module and the subprocess module. Read on and pick whichever fits you.

Using the os module

First up is the os module and its system() method. Per the official documentation, the os module gives you a portable way to use operating-system-dependent functionality. Handy. To run a Linux command with it, write the code like this:

Sample Code using system()

import os
os.system('pwd')
os.system('cd ~')
os.system('ls -la')

Those four lines check your current directory, move you to your home directory, and list everything in detail. Simple enough, but there’s a catch. With system(), you can’t store the resulting output in a variable.

The other option is the popen() method, still part of the os module. It opens a pipe to or from the command line. A pipe connects one command’s output to another command’s input, which makes it reachable inside Python. Storing the output as a variable with popen() looks like this:

Sample code using popen()

import os
stream = os.popen('ls -la')
output = stream.readlines() 

Print the stream variable and you’ll see its return data: the actual commands run, the mode, and the address. Want the whole output as one string instead? Swap readlines() for read().

Using the subprocess module

The second way to run Linux commands from Python is the newer subprocess module. This module lets you spawn new processes, connect to their input/output/error pipes, and grab their return codes. It was built to replace both os.system() and os.popen(). The one method that matters in subprocess is run(). With it you can do everything above and more, just with different arguments. Use this as reference:

Writing a simple command using subprocess

import subprocess subprocess.run('ls')

Called like that, it runs the ls command in your terminal. Unlike os.system(), it won’t work if you tack on a switch and pass the whole thing as one string like subprocess.run('ls -la'). That behavior is on purpose: the method handles quoting and escaping for you, so formatting mistakes don’t turn into errors. To run ls -la, pass the command as a list: subprocess.run(['ls','-la'). Or set the shell argument to True to pass the whole thing as a string. Just know that route can be a security risk if you’re feeding it untrusted input.

Writing a command with switches

import subprocess x = subprocess.run(['ls', '-la'])

import subprocess subprocess.run(['ls -la'], shell=True)

To stash the command output in a variable, just assign it like any other data. Heads up though, the result won’t be what you’d expect. The whole point of run is to execute the shell command inside Python, so what you get back isn’t the terminal output. It’s the return data, same as with os.open. Check it with the code below.

Storing the command output to a variable

import subprocess x = subprocess.run(['ls', '-la'])
print(x)
print(x.args)
print(x.returncode)
print(x.stdout)
print(x.stderr)

This one breaks down the return data of your command using the method’s arguments. Here are the ones you’ll reach for most:

• args – returns the actual commands executed.
• returncode – returns the return code of the output. 0 means no error.
• stdout – captured stdout from the child process.
• stderr – captured stderr stream from the child process.

Since we didn’t capture the output in the last snippet, both stdout and stderr come back as None. To turn on the capture output argument, use this:

import subprocess x = subprocess.run(['ls', '-la'], capture_output=True)

Print x now and you’ll get the list of items in your current directory, as type bytes. Turn it into a string with x.stdout.decode(). Or pass text=True to the main function. Either way the output should now match what you see in the terminal.

Last thing: let’s run a Linux command from Python and save the terminal output straight into a text file. Easy with subprocess. Redirect the stdout stream to your text file with the stdout argument.

Saving the command output to a text file

import subprocess with open('list.txt', 'w') as f: subprocess.run(['ls','-la'], stdout=f)

And that’s the tutorial. For quick one-off scripts, os.system() and os.popen() get the job done. For anything bigger, reach for the subprocess module instead.