Semgrep tutorial: scanning code and writing custom rules for insecure patterns

What Semgrep does

Semgrep is a static analysis tool that matches code patterns that look like the code itself. Rules are written in YAML with a pattern using metavariables such as $X and the ... operator for “anything here”. It supports many languages and runs locally or in CI.

Snippet

pip install semgrep
semgrep scan --config p/python .
semgrep scan --config p/secrets --config p/owasp-top-ten .

A custom rule

The rules below flag subprocess calls with shell=True whose command is not a string literal, and requests calls that disable certificate verification. pattern-not excludes the safe literal case.

Snippet

rules:
  - id: subprocess-shell-true-with-variable
    languages: [python]
    severity: ERROR
    message: subprocess with shell=True and a non-literal command can allow command injection; pass an argument list instead.
    patterns:
      - pattern: subprocess.$FUNC($CMD, ..., shell=True, ...)
      - pattern-not: subprocess.$FUNC("...", ..., shell=True, ...)
  - id: requests-verify-false
    languages: [python]
    severity: WARNING
    message: TLS certificate verification is disabled.
    pattern: requests.$METHOD(..., verify=False, ...)

Lab: write, test, and run in CI

Save the rules as rules.yml. Write a test file with one positive and one negative example of each and annotate them with ruleid and ok comments, then run semgrep --test. Scan one of your own script repositories with the rules and the community Python pack, fix real findings, and add a CI job that runs semgrep scan --config rules.yml --error.

Snippet

cat > test_rules.py <<'EOF'
import subprocess, requests
host = input()
# ruleid: subprocess-shell-true-with-variable
subprocess.run(f'ping {host}', shell=True)
# ok: subprocess-shell-true-with-variable
subprocess.run('uptime', shell=True)
# ruleid: requests-verify-false
requests.get('https://lab.internal', verify=False)
EOF
semgrep --test --config rules.yml test_rules.py
semgrep scan --config rules.yml --error .

Tuning

Start with a few high-confidence rules so developers trust the results. Use nosemgrep comments with a reason for justified exceptions. Track findings over time rather than trying to fix everything at once in a large codebase.

Diagram

Semgrep pattern operators

Patterns look like code.

$X
Metavariable: matches any expression and binds it.
...
Any sequence of arguments or statements.
patterns + pattern-not
Match A but not B.
pattern-either
Match any of several patterns.
metavariable-regex
Constrain a metavariable with a regex.

Check yourself

1. What does ... mean in a Semgrep pattern?

2. How do you exclude a safe variant from a rule?

3. How do you verify a rule matches what you intend?