How to Execute Shell Commands Using Python

Python is a popular choice for shell scripting and task automation that even Google uses Python in its online courses for Automation. It is also popular in system administration because it can execute shell commands using only its default libraries. In this tutorial, you will learn just that – to run Linux shell commands with Python using a Raspberry Pi as a host computer.

Things you need:

• Computer
• Basic knowledge in using CLI (Command-Line Interface)

If you don’t know what CLI is or know it, but you need a refresher, kindly visit this tutorial first.

Introduction

Automation is a buzzword hot as your processor running at 100% in a room with no AC. Home management, data extraction, those omniscient talking boxes in the kitchen, and even self-driving cars! They are all products of automation. Well, it makes sense since we’re living in the fourth industrial revolution. Heard of it? It is where humans and machines try to work as one, and it is happening now.

Using a Raspberry Pi board opens a lot of doors in automation. With its GPIO, you can interface the credit-card sized computer into almost anything that throws or receives digital data. And since it is a fully-fledged Linux computer, you can automate your computer tasks as well. What’s more, Python makes it unimaginably easier! There are two ways to run Linux commands with Python: using the os module and using the subprocess module. Read further and see what you prefer.

Using the os module

First, is the os module using its system() method. According to official documentation, the os module provides a portable way of using operating system dependent functionality. Ain’t that convenient? To use it to run a Linux command, you must write the code like this:

Sample Code using system()

import os
os.system('pwd')
os.system('cd ~')
os.system('ls -la')

This 4-liner checks your current directory, change location to your home directory, and lists all the contents in detail. It’s a pretty straightforward implementation, but there’s a downside. With system(), you are not allowed to store the resulting output as a variable.

Alternatively, you can use the popen() method – still under the os module. It opens a pipe from or to the command line. A pipe connects a command’s output to another command’s input. This makes it accessible within Python. To use popen() to store a as a variable, it should be similar to this:

Sample code using popen()

import os
stream = os.popen('ls -la')
output = stream.readlines()

If you print the stream variable, you will see its return data. This consists of the actual commands executed, the mode, and the address. Furthermore, if you want to get the whole output as one string, change readlines() to read().

Using the subprocess module

The second way to run Linux commands with Python is by using the newer subprocess module. This module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. It was created to replace both os.system() and os.popen() functions.The only method that matters in subprocess is run(). With it, you can do everything we’ve done above and more using different arguments. Use the following code as reference:

Writing a simple command using subprocess

import subprocess

subprocess.run('ls')

Using the method like this will execute the command ls in your terminal. Unlike os.system(), it doesn’t work when you add a switch and enter it fully like subprocess.run('ls -la'). This feature allows the method to take care of quoting and escaping problems hence preventing errors from formatting. To execute ls -la, pass the command as a list: subprocess.run(['ls','-la'). Alternatively, you can make the shell argument True to pass the whole thing as a string. Just take note that this can be a security risk if you’re using untrusted input.

Writing a command with switches

import subprocess

x = subprocess.run(['ls', '-la'])

import subprocess

subprocess.run(['ls -la'], shell=True)

Next, to store the command output inside a variable, simply do it just like any other data. The result won’t be what you’re expecting, however. Since the main purpose of run is to execute the shell command within python, the result won’t be the output you see in the terminal. It will be the return data just like in os.open. You can check it using the code below.

Storing the command output to a variable

import subprocess

x = subprocess.run(['ls', '-la'])
print(x)
print(x.args)
print(x.returncode)
print(x.stdout)
print(x.stderr)

This sketch dissects the return data of your command using the method’s arguments. Here are some of the frequently used ones:

• args – returns the actual commands executed.
• returncode – returns the return code of the output. 0 means no error.
• stdout – captured stdout from the child process.
• stderr – captured stderr stream from the child process.

Since we did not capture the output in the previous code, we will get None with both stdout and stderr arguments. To enable the capture output argument, refer to the following code:

import subprocess

x = subprocess.run(['ls', '-la'], capture_output=True)

Now, if you print x, you will get the list of items in your current directory of type bytes. Convert it to a string by writing x.stdout.decode(). Alternatively, you can pass the argument text=True with the main function. The output now should look exactly the same as what you have in the terminal.

Ultimately, we will run Linux commands with Python and save the output you see in the terminal into a text file – a simple task with subprocess. You just need to redirect the stdout stream to your text file using the argument stdout.

Saving the command output to a text file

import subprocess

with open('list.txt', 'w') as f:
    subprocess.run(['ls','-la'], stdout=f)

That’s it for this tutorial! To summarize, for quick execution of a short script, consider using os.system() and os.popen() methods. But if you have a more comprehensive script to run, you might want to use the subprocess module instead.