Of course! Let's break down how to build a Python Low-Level Discovery (LLD) rule for Zabbix. This is a powerful technique for automatically monitoring new or dynamic resources without manual configuration.

We'll use a very common and practical example: discovering network interfaces on a Linux server.

What is Zabbix LLD?

Low-Level Discovery allows Zabbix to automatically:

1. Find resources on a host (e.g., network interfaces, mounted filesystems, Docker containers).
2. Create new items, triggers, and graphs for each discovered resource.
3. Monitor these resources automatically.

Instead of manually creating an item for eth0, eth1, lo, etc., you create one LLD rule that finds all interfaces and then generates a template for each one.

The Big Picture: How Python & LLD Work Together

The process flows like this:

1. Zabbix Agent: The Zabbix agent is configured to run a user parameter.
2. User Parameter: This parameter points to a Python script on the monitored machine.
3. Python Script: The script executes, performs the discovery (e.g., lists network interfaces), and formats the output as a JSON object.
4. Zabbix Server: The server receives this JSON data.
5. LLD Rule: The LLD rule on the Zabbix server is configured to parse this JSON.
6. Prototypes: The server uses the parsed data to create real monitoring items, triggers, etc., based on "prototypes" you've defined.

Step 1: The Python Discovery Script

This is the core of our solution. The script's job is to find the resources and return a JSON object that Zabbix can understand.

For our example, we'll create a script that lists all network interfaces, excluding the loopback (lo) interface.

File: /etc/zabbix/scripts/discover_network_interfaces.py

#!/usr/bin/env python3
import subprocess
import json
import sys
def get_network_interfaces():
    """
    Discovers network interfaces on a Linux system using 'ip' command.
    Excludes the loopback interface.
    """
    try:
        # Use the 'ip' command, which is more modern than 'ifconfig'
        # -o: output in one line
        # -j: output in JSON format (easiest to parse!)
        result = subprocess.run(['ip', '-j', '-o', 'link'], capture_output=True, text=True, check=True)
        interfaces = []
        data = json.loads(result.stdout)
        for item in data:
            ifname = item['ifname']
            # Exclude the loopback interface
            if ifname == 'lo':
                continue
            interfaces.append({
                "{#IFNAME}": ifname
            })
        return interfaces
    except subprocess.CalledProcessError as e:
        print(f"Error executing command: {e}", file=sys.stderr)
        return []
    except json.JSONDecodeError as e:
        print(f"Error parsing JSON: {e}", file=sys.stderr)
        return []
    except Exception as e:
        print(f"An unexpected error occurred: {e}", file=sys.stderr)
        return []
if __name__ == "__main__":
    discovered_interfaces = get_network_interfaces()
    print(json.dumps(discovered_interfaces, indent=4))

Key points about the script:

• Shebang (#!/usr/bin/env python3): Ensures it runs with Python 3.
• subprocess.run: Executes the ip command securely.
• ip -j -o link: This is the magic. It outputs network link information in a clean, machine-readable JSON format, avoiding complex parsing.
• Exclusion Logic: We explicitly skip the lo interface.
• JSON Output Format: The script must output a JSON array of objects. Each object represents one discovered item.
• {#IFNAME}: This is a macro. Zabbix uses this to uniquely identify each discovered item. You can name it whatever you want (e.g., {#INTERFACE_NAME}), but be consistent.

Make the script executable:

sudo chmod +x /etc/zabbix/scripts/discover_network_interfaces.py

Step 2: Configure the Zabbix Agent

Now, tell the Zabbix agent to run this script. We do this by adding a UserParameter to the agent's configuration file.

File: /etc/zabbix/zabbix_agentd.conf

# Add this line at the end of the file
UserParameter=net.if.discovery,/etc/zabbix/scripts/discover_network_interfaces.py

Explanation:

• net.if.discovery: This is the unique key we will use in Zabbix to reference the discovery data.
• /etc/zabbix/scripts/discover_network_interfaces.py: The command to execute.

Restart the Zabbix agent:

sudo systemctl restart zabbix-agent

You can test this from the Zabbix server:

# Replace <zabbix_agent_ip> with the IP of your target host
zabbix_get -s <zabbix_agent_ip> -p 10050 -k "net.if.discovery"

You should see the JSON output from your Python script.

Step 3: Configure the LLD Rule in Zabbix GUI

Now, log in to your Zabbix frontend and configure the discovery rule.

1. Navigate to the Host: Go to Configuration -> Hosts and select the host you want to configure.
2. Create Discovery Rule:
   • Go to the Discovery tab.
   • Click Create discovery rule.
   • Name: Network interfaces discovery
   • Type: Zabbix agent
   • Key: net.if.discovery (This must match the UserParameter key exactly).
   • Update interval: 1h (or whatever you prefer).
   • Custom checks: Leave unchecked.
   • Click Add and then Save.

At this point, Zabbix will run the discovery. You can check the results under Monitoring -> Discovery. You should see your host and the number of "Found" interfaces.

Step 4: Create Item, Trigger, and Graph Prototypes

Prototypes are blueprints. Zabbix uses them to create the actual monitoring elements for each discovered interface.

1. Back on the host's Discovery tab, click on the Network interfaces discovery rule you just created.
2. You will see tabs for Items, Triggers, and Graphs. We'll create one of each.

A) Item Prototype

This tells Zabbix what to monitor for each interface.

• Go to the Items tab and click Create item prototype.
• Name: Network traffic on {#IFNAME}
• Key: net.if.in[{#IFNAME}] (This is a standard Zabbix key for interface traffic. Zabbix will substitute {#IFNAME} with eth0, eth1, etc.)
• Type: Zabbix agent
• Delay: 30s (or your preferred interval).
• History: 90d
• Click Add to create the item for outgoing traffic.
• Now, create another item prototype for outgoing traffic:
  • Name: Network traffic on {#IFNAME} (outgoing)
  • Key: net.if.out[{#IFNAME}]
  • (Use the same settings as above).
• Click Save.

B) Trigger Prototype

This tells Zabbix when to generate an alert.

• Go to the Triggers tab and click Create trigger prototype.
• Name: High network traffic on {#IFNAME}
• Expression: Click Add and build the following expression:

  {<your_host_name>:net.if.in[{#IFNAME}].avg(5m)} > 1000000

  This expression means: "If the average incoming traffic on interface {#IFNAME} over the last 5 minutes is greater than 1,000,000 bytes (1 MB/s), then fire the trigger."

• Severity: Warning or Average.
• Click Add to create a trigger for outgoing traffic if you wish.
• Click Save.

C) Graph Prototype

This provides a visual representation of the data.

• Go to the Graphs tab and click Create graph prototype.
• Name: Network traffic on {#IFNAME}
• Width: 900
• Height: 200
• Under Graph items, click Add.
  • Item: Select the incoming traffic item prototype (Network traffic on {#IFNAME}).
  • Type: Normal
  • Y axis: Left
  • Draw style: Area
  • Color: Choose a color (e.g., blue).
• Click Add again for the outgoing traffic.
  • Item: Select the outgoing traffic item prototype (Network traffic on {#IFNAME} (outgoing)).
  • Y axis: Left
  • Draw style: Line
  • Color: Choose another color (e.g., green).
• Click Save.

Step 5: Verify and Enjoy!

After a few minutes (depending on your update interval), Zabbix will have created the actual items, triggers, and graphs for each network interface it discovered. You can see them under the host's Items, Triggers, and Graphs tabs, now prefixed with the discovered interface name (e.g., Network traffic on eth0).

If you plug in a new USB-to-Ethernet adapter, Zabbix will automatically discover the new interface (e.g., eth1) and start monitoring it without any manual intervention.