SSM

Run Command Troubleshooting Checklist

---

• Verify if the instance is online (SSM agent pre-requisites). Is the Agent up to date?
• Verify if SSM document supports the type of instance you want to target (Windows and Linux)
• Ask for command ID and verify the status in SSM K2 page from SSM dashboard or from Admiral page.
• Check the logs and the stdout location:
  • The agent also stores the script (_script.ps1) and outputs (stderr and stdout) in the worker (Orchestration) folders located at:
    • /var/lib/amazon/ssm/<instanceID>/document/orchestration/<RunCommandId>/plug-in/step_number.plugin>/std*

/var/lib/amazon/ssm/< i-xxxxxxxxxxxx>/document/orchestration/<command ID>/awsrunShellScript/[PatchLinux or runInSpecLinux for example]/stdout

/var/lib/amazon/ssm/<i-instanceid>/document/orchestration/<command ID>/awsrunShellScript/[PatchLinux or runInSpecLinux for example]/stderr

# example: "runInSpecLinux"
/var/lib/amazon/ssm/i-0bb46bc5e41a7a36b/document/orchestration/6b141723-0c4b-4726-8770-bc8eed816007/awsrunShellScript/runInSpecLinux/stdout

# If the command action is  aws:runCommand :
/var/lib/amazon/ssm/<i-instanceid>/document/orchestration/<command ID>/awsrunDocument/<stepName>/std*

• Manual execution from instance?
• Check the Run command advanced playbooks.
• Reproduce issue
• Cut a TT/SIM

Logs

---

• /var/log/amazon/ssm

You can view SSM Agent logs on instances in the following locations:

/var/log/amazon/ssm/amazon-ssm-agent.log 

/var/log/amazon/ssm/errors.log

/var/log/amazon/ssm/audits/amazon-ssm-agent-audit-YYYY-MM-DD

---

Checking command logs:

here is path (replace instance id and command id)

/var/lib/amazon/ssm/<i-instanceid>/document/orchestration/<command ID>/awsrunShellScript/PatchLinux/stdout

/var/lib/amazon/ssm/<i-instanceid>/document/orchestration/<command ID>/awsrunShellScript/PatchLinux/stderr

# If the command action is  aws:runCommand :
/var/lib/amazon/ssm/<i-instanceid>/document/orchestration/<command ID>/awsrunDocument/<stepName>/std*

How to install SSM Agent

---

Manually install SSM Agent on EC2 instances for Linux https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html

How to reboot managed instance once by using custom Run Command document

---

Yaml template:

--- 
description: "Command Document Example JSON Template"
mainSteps: 
  - 
    action: "aws:runShellScript"
    inputs: 
      runCommand: 
        - |-
            #!/bin/bash
            REBOOT_INDICATOR=/var/lib/amazon/ssm/ssm-reboot-indicator
            echo 'Rebooting the host via SSM'
            if [ ! -f ${REBOOT_INDICATOR} ]; then
              uptime
              echo 'rebooting... as the reboot file does not exist'
              touch ${REBOOT_INDICATOR}
              exit 194
            else
              echo 'reboot file exist'
              ls -l ${REBOOT_INDICATOR} 
              uptime 
              rm -f ${REBOOT_INDICATOR}
              exit 0
            fi
    name: example
schemaVersion: "2.2"

• It first checks if the reboot tracker file (/var/lib/amazon/ssm/ssm-reboot-indicator) exists in the 'if' condition. If it does not, then it will create it, run mkfs, and then reboot it by sending out exit 194.

• Once the system reboots, then it will go through the entire script again. Now since the reboot tracker does exist, it will remove the file and return exit 0.

• The use of this if-else statement and the reboot tracker file helps prevent the instance from going into a reboot-loop.

--- 
description: "Command Document Example JSON Template"
mainSteps: 
  - 
    action: "aws:runShellScript"
    inputs: 
      runCommand: 
        - |-
            #!/bin/bash
            REBOOT_INDICATOR=/var/lib/amazon/ssm/ssm-reboot-indicator
            echo 'Mounting volume and Rebooting the host via SSM'
            mount -a
            if [ ! -f ${REBOOT_INDICATOR} ]; then
              uptime
              echo 'Creating filesystem on /dev/nvme1n1'
              mkfs -t ext4 /dev/nvme1n1
              echo 'rebooting... as the reboot file does not exist'
              touch ${REBOOT_INDICATOR}
              exit 194
            else 
              echo 'reboot file exist, the server has been rebooted after mounting'
              ls -l ${REBOOT_INDICATOR}
              uptime
              rm -f ${REBOOT_INDICATOR}
              exit 0
            fi
    name: example
schemaVersion: "2.2"