Of course! Using subprocess.Popen to capture stdout is a fundamental and powerful technique in Python for running external commands and interacting with their output. It's more flexible and modern than the older os.system() or os.popen() functions.

Here’s a complete guide, from the basics to more advanced use cases.

The Short Answer: The Best Way (Python 3.5+)

For most modern use cases, the recommended approach is to use the subprocess.run() function, which is a high-level wrapper around Popen. It's simpler and less error-prone.

import subprocess
# The command you want to run, as a list of strings
command = ["ls", "-l"]
# Run the command and capture the output
try:
    # text=True decodes stdout/stderr using default encoding (usually utf-8)
    # check=True raises an exception if the command returns a non-zero exit code
    # capture_output=True captures stdout and stderr
    result = subprocess.run(command, capture_output=True, text=True, check=True)
    # Print the captured standard output
    print("Command succeeded!")
    print("STDOUT:")
    print(result.stdout)
except subprocess.CalledProcessError as e:
    print("Command failed!")
    print("Return code:", e.returncode)
    print("STDOUT:")
    print(e.stdout)
    print("STDERR:")
    print(e.stderr)

The Core of Your Question: Using Popen Directly

Popen (Process Open) gives you fine-grained control. You start the process and then manage it yourself, which is essential for long-running commands or when you need to send data to the process's stdin.

Basic Example: Capturing stdout

The key is to pass a PIPE object to the stdout argument of Popen. This tells Python to create a pipe that you can read from.

import subprocess
# The command to run
command = ["echo", "Hello from the subprocess!"]
# 1. Create the Popen object
#    - stdout=subprocess.PIPE: This is the crucial part. It creates a pipe for stdout.
#    - text=True: Decodes stdout/stderr as text (avoids dealing with bytes).
#    - encoding='utf-8': Explicitly sets the encoding.
process = subprocess.Popen(command, stdout=subprocess.PIPE, text=True, encoding='utf-8')
# 2. Communicate with the process
#    - .communicate() sends data to stdin (if any) and reads all data from stdout and stderr.
#    - It waits for the process to terminate, which is important.
#    - It returns a tuple: (stdout_data, stderr_data)
stdout, stderr = process.communicate()
# 3. Check the return code (exit status) of the process
#    A return code of 0 usually means success.
if process.returncode == 0:
    print("Command executed successfully.")
    print("Captured STDOUT:")
    # The output from .communicate() will have a trailing newline, so we strip it
    print(stdout.strip())
else:
    print(f"Command failed with return code {process.returncode}")
    if stderr:
        print("STDERR:")
        print(stderr.strip())

Key Popen Arguments Explained

Advanced Scenarios

Real-time Output (Streaming)

.communicate() reads all output at once and waits for the process to finish. If you have a long-running command (like ping or a server log), you want to print output as it arrives.

To do this, you must use process.stdout.readline() in a loop.

Important: You must also ensure that the process's stdout is line-buffered. Some programs (like Python scripts) might not flush their output buffer until the process exits, causing your script to hang. Using sys.stdout.flush() in the subprocess can help.

import subprocess
import time
# A command that produces output over time
command = ["ping", "-c", "5", "example.com"]
print("Starting process and streaming output...")
process = subprocess.Popen(
    command,
    stdout=subprocess.PIPE,
    text=True,
    encoding='utf-8'
)
try:
    # Loop as long as the process is running
    while process.poll() is None:  # poll() returns None while the process is running
        # readline() reads one line at a time from the pipe
        line = process.stdout.readline()
        if line:
            print(line.strip()) # Print the line without the trailing newline
    # Read any remaining output after the process has finished
    remaining_output = process.stdout.read()
    if remaining_output:
        print(remaining_output.strip())
finally:
    # Ensure the process is terminated if the script is interrupted
    if process.poll() is None:
        print("Process did not terminate cleanly. Killing it.")
        process.kill()
    process.wait() # Wait for the process to be fully reaped
print("Process finished.")

Handling Both stdout and stderr

You can capture both streams simultaneously. The communicate() method makes this easy.

import subprocess
# A command that will produce output on both stdout and stderr
command = ["bash", "-c", "echo 'This is standard output'; echo 'This is standard error' >&2"]
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# communicate() returns a tuple: (stdout_data, stderr_data)
stdout, stderr = process.communicate()
print(f"Return Code: {process.returncode}")
print("-" * 20)
print("STDOUT:")
print(stdout)
print("-" * 20)
print("STDERR:")
print(stderr)

Sending Data to stdin

If a command accepts input from stdin, you can send it data using the input argument of .communicate().

import subprocess
# A command that reads from stdin and writes to stdout
command = ["sort"] # sort will sort lines of text you give it
# The data to send to the command's stdin
input_data = "banana\napple\ncherry\n"
process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
# The input argument of communicate sends the data to the process's stdin
# and then closes the pipe, signaling that there's no more input.
stdout, stderr = process.communicate(input=input_data)
print("Command executed.")
print("Sorted output:")
print(stdout)

Summary: Popen vs. run vs. call