Semgrep Architecture: Comprehensive Reference

Semgrep (Semantic Grep) is a multi-language static analysis tool that matches code by its structure – not just its text – using a unified Abstract Syntax Tree and a rich pattern language. This document covers the full internal architecture from CLI entry-point to taint sink detection.

Table of Contents

0. Introduction
   • What is SAST?
   • History of Semgrep
1. High-Level Architecture
2. Component Breakdown
3. The Full Analysis Pipeline
4. Target Discovery & Filtering
5. Rule Parsing & Optimization
6. Parsing & the Universal AST
7. The Matching Engine
8. The Intermediate Language (IL) & CFG
9. Taint Analysis (Dataflow)
10. Output & Reporting Pipeline
11. OSemgrep / RPC Architecture
12. Key Data Structures

-1. Why?

I have been working as a DevSecOps engineer for nearly four years now. When I started, I had little exposure to tooling like SAST, SCA, or DAST. My mindset was firmly rooted in offensive security. Penetration testing was the goal, the dream.

Then I discovered SAST. My first reaction was genuine excitement: “This is incredible, automated code analysis at scale!” But over time, reality set in. In practice, most findings tend to be noisy. False positives are plentiful, and the truly critical vulnerabilities are rarely surfaced by off-the-shelf rules. To be clear: I am not saying these tools are useless, far from it. They are an essential layer of defense in any mature security program. But they do have limits.

That tension, between the promise and the reality of SAST, planted a question in my head that would not go away: why not build my own? The idea kept nagging at me until I finally decided to dig in. And what a rabbit hole it turned out to be.

The natural starting point was Semgrep. Understanding how it works under the hood, the parsing pipeline, the universal AST, the matching engine, the taint analysis, has been one of the most intellectually rewarding deep-dives I have done in a long time. This article is the result of that exploration.

There will be more articles on this topic. I have a lot to say about static analysis, and I am just getting started.

No em dashes used anywhere. You may need to reload the file in your editor to see the changes.

0. Introduction

0.1 What is SAST?

Static Application Security Testing (SAST) is a white-box security technique that analyzes source code, bytecode, or binary code without executing the program. A SAST tool reads the code the same way a compiler or interpreter would, builds an internal model of program structure and data flow, and then applies a library of security rules to that model to identify vulnerabilities.

SAST sits at the earliest possible stage of the Software Development Lifecycle (SDLC). This “shift-left” principle means that bugs are caught before they can ever reach a staging or production environment – dramatically reducing the cost and effort of fixing them.

Core properties of a SAST tool:

SAST vs. Other Security Testing Approaches

The four major security testing categories differ along two axes: whether source code access is needed, and whether the application must be running.

Modern security programs use all of these layers together. SAST is the first line of defense because it is the cheapest and fastest to run.

How SAST Works: The Core Technique

Most SAST tools follow the same fundamental pipeline:

Semgrep is unique among SAST tools because it exposes this pipeline to end-users directly. Instead of shipping only a fixed rule library, Semgrep lets security engineers write custom rules in YAML using a pattern language that mirrors actual code syntax – no formal language theory required.

0.2 History of Semgrep

Semgrep was not built from scratch. Its roots trace back to research done inside Facebook more than fifteen years ago.

The pfff Lineage (2009)

In 2009, Yoann Padioleau – then an engineer at Facebook – created pfff (“Parsing Framework for Fun”). pfff was an OCaml library and toolset that could parse multiple programming languages (PHP, C, OCaml, Java) into a common AST representation. The goal was to build code navigation, refactoring, and lightweight static analysis tools that worked uniformly across Facebook’s polyglot codebase.

pfff introduced two foundational ideas that Semgrep still relies on today:

1. A universal AST (AST_generic) shared across all supported languages.
2. sgrep – a prototype “semantic grep” that let engineers write structural code patterns (with wildcards) that matched against the pfff AST instead of raw text.

From sgrep to Semgrep (2013-2019)

The sgrep prototype demonstrated that structural matching was practical and useful for finding bugs at scale inside Facebook. The concept was refined internally over several years but not yet released publicly. The OCaml core continued to evolve alongside pfff.

Open Source Launch (2020)

In 2020, r2c (Return to Corporation, later renamed Semgrep Inc.) spun the technology out and released Semgrep as a free, open-source tool under the LGPL-2.1 license. The release included:

• A Python CLI wrapping the OCaml binary.
• The Semgrep Registry – a community-maintained library of security rules.
• Support for 10+ languages from day one.

This release was a breakthrough: for the first time, any developer or security engineer could write structural code patterns without needing a PhD in program analysis.

Expansion and Commercialization (2021-2022)

Semgrep Inc. began building commercial capabilities on top of the OSS core:

• Semgrep Pro / DeepSemgrep (2021): Added inter-procedural taint analysis and cross-file dataflow – tracking how tainted data flows not just within a function but across module boundaries and entire codebases.
• Semgrep Supply Chain / SCA (2022): Integrated dependency manifest scanning to detect known CVEs in third-party libraries, tying reachability analysis (“is this vulnerable function actually called?”) back to the SAST engine.
• Semgrep Secrets (2022-2023): Extended the rule language to detect hardcoded credentials and API keys, with entropy scoring and AI-powered online validation.

OSemgrep and Modern Semgrep (2023-present)

As the Python CLI became a performance and maintenance bottleneck, Semgrep Inc. began the OSemgrep project: a full reimplementation of the Python CLI in OCaml, merging the CLI and core engine into a single binary. This eliminates subprocess overhead, enables tighter integration with the IDE via RPC and LSP protocols, and simplifies deployment.

1. High-Level Architecture

Semgrep is split into two tiers: a Python CLI that handles user-facing concerns, and an OCaml core engine that performs all heavy lifting - parsing, matching, and dataflow analysis.

2. Component Breakdown

2.1 Semgrep CLI (cli/src/semgrep/)

2.2 Semgrep Core (src/)

3. The Full Analysis Pipeline

4. Target Discovery & Filtering

4.1 File Discovery Flow

4.2 Language Detection (Guess_lang.ml)

Detection is multi-layered (in priority order):

1. Explicit --lang flag – overrides all heuristics
2. File extension – .py → Python, .ts → TypeScript, etc.
3. Shebang line (#!/usr/bin/env python3)
4. .semgrepignore / rule paths: selectors

4.3 The .semgrepignore System (Semgrepignore.ml)

The .semgrepignore layering: built-in defaults < repo .semgrepignore < .gitignore < CLI --exclude/--include flags. Later rules win; ! prefix negates.

5. Rule Parsing & Optimization

5.1 Rule YAML Structure

A Semgrep rule contains:

• id - unique identifier
• languages - target language(s)
• pattern / patterns / pattern-either - the formula
• pattern-not / pattern-not-inside - negative clauses
• metavariable-* - metavariable constraints (regex, type, comparison, pattern)
• mode - search (default) or taint
• fix - optional autofix template

5.2 Rule Parsing Pipeline

5.3 Formula Algebra

Rules can express boolean logic over patterns:

5.4 Prefiltering (src/prefiltering/)

Before any parsing occurs, rules are statically analyzed to extract “must-have” literal strings. A bloom-filter-like scan of raw file bytes allows the engine to skip files that provably cannot match any active rule - often eliminating 90%+ of files without touching the parser.

Prefiltering: Rules are statically analyzed to extract mandatory string literals. Files not containing any of those literals are skipped before parsing – often eliminating 90%+ of the corpus.

6. Parsing & the Universal AST

6.1 Why a Universal AST?

Semgrep supports 30+ languages using a single matching engine by normalising every language’s AST into AST_generic - a lingua franca for code structure. Adding a new language means writing only a parser that outputs this type; the matching logic is reused automatically.

6.2 Parser Backends

6.3 AST Normalisation & Naming

After raw parsing, two passes enrich the generic AST:

1. Normalisation (Normalize_generic.ml): Canonicalise syntactic sugar (e.g., x += 1 → x = x + 1).
2. Name Resolution (Naming_AST.ml): Resolve variable references to their definition scope. This enables the engine to distinguish os.path.join (the stdlib function) from a local path.join.

6.4 Pattern Parsing

Rule patterns go through the same parsers but with a dedicated entry point (Parse_pattern.ml). Metavariable tokens ($X, $...ARGS) are injected as special AST nodes before parsing so the grammar treats them as valid expressions - then the engine interprets them during matching.

7. The Matching Engine

7.1 Overall Matching Architecture

7.2 Pattern_vs_code.ml - The Core Matcher

This is the largest file in the codebase (~155 KB). It implements a recursive descent structural comparator between a pattern AST node and a code AST node using OCaml’s Monadic_bind pattern.

Key matching rules:

7.3 Metavariable Constraints (Post-match filters)

After a raw structural match, the engine evaluates metavariable constraint clauses in sequence. All must pass for the finding to be reported.

7.4 Matching Visitor (Matching_visitor.ml)

The visitor drives the match attempt over every node in the code AST. For each rule formula it:

1. Walks the code AST top-down.
2. Attempts the pattern at each node.
3. Collects Range_with_metavars - the code range that matched alongside metavar bindings.
4. Applies boolean formula operators (AND, OR, NOT, INSIDE) to combine raw matches.

8. The Intermediate Language (IL) & CFG

Search-mode matching works directly on the AST. Taint mode requires understanding execution order, so the engine first lowers the AST into an Intermediate Language (IL) and then builds a Control Flow Graph (CFG).

8.1 AST → IL (AST_to_IL.ml)

The IL simplifies dataflow by making all side effects explicit as discrete instructions, each with a clear left-hand side (assignment target) and right-hand side (expression), eliminating complex nested expressions.

8.2 CFG Construction (CFG_build.ml)

8.3 Constant Propagation (Constant_propagation.ml)

Before taint analysis, the engine runs sparse constant propagation over the CFG. This resolves:

• String literals assigned to variables (url = "http://evil.com")
• Simple arithmetic on integer constants
• Concatenation of known string constants

This enriches the IL so taint patterns can match the effective value of expressions, not just their syntactic form.

9. Taint Analysis (Dataflow)

9.1 End-to-End Taint Flow

9.2 Fixpoint Engine (Dataflow_core.ml)

The core dataflow algorithm is a worklist-based forward analysis:

9.3 Taint Representation (Taint.ml / Taint_shape.ml)

Taint is not just a boolean - it carries provenance information:

9.4 Taint Propagation Rules

Taint Propagation Rules: x = tainted_y taints x; x.f = tainted_y taints field x.f; call return is tainted if a propagator rule applies; string interpolation of tainted data taints the result; sanitizer calls remove taint.

9.5 Source / Sink / Sanitizer Matching (Taint_spec_match.ml)

The “best match” strategy prevents double-reporting:

Best Match Strategy (Taint_spec_match.ml): When multiple overlapping AST nodes match a source/sink pattern (e.g., foo vs foo.bar), the engine keeps only the longest/most-specific match to avoid double-reporting.

9.6 Intra- vs Inter-procedural Analysis

10. Output & Reporting Pipeline

10.1 Core Output → CLI Formatting

10.2 Exit Codes

10.3 The # nosemgrep Suppression System

# nosemgrep suppression: If a finding is on line N and line N or N-1 contains # nosemgrep (optionally followed by a colon-separated list of rule IDs), the finding is suppressed and not reported.

11. OSemgrep / RPC Architecture

11.1 OSemgrep (src/osemgrep/)

OSemgrep is the OCaml reimplementation of the Python CLI - a long-term effort to bring the full CLI into the OCaml binary, eliminating the Python dependency for performance-critical or embedded scenarios.

OSemgrep vs pysemgrep: The Python CLI invokes semgrep-core as a subprocess and exchanges JSON over stdin/stdout. OSemgrep (the OCaml reimplementation) calls core directly as a library, eliminating the subprocess boundary and the Python runtime dependency.

11.2 RPC / LSP Bridge (src/rpc/, cli/src/semgrep/rpc.py)

For IDE integrations, Semgrep exposes a JSON-RPC interface:

RPC / LSP Bridge: The IDE sends LSP events to the language server (rpc.py / lsp_legacy/). The server forwards them as JSON-RPC scan requests to the Core. Findings are returned as LSP diagnostics and shown as inline annotations in the editor.

12. Key Data Structures

12.1 AST_generic Node Hierarchy (simplified)

The any union covers all node kinds. The most common sub-types:

12.2 Rule.t (OCaml Record, simplified)

12.3 Range_with_metavars.t (Match Result)

Appendix A: Parser Coverage

Appendix B: Glossary

See also

• Code auditing 101