← research

oraios/serena — how a best-in-class coding MCP is actually built

Jul 14, 2026

Target: oraios/serena — 26k-star "MCP toolkit for coding: semantic retrieval and editing." Method: read the real implementation, not the README. All paths below are under `/Developer/reference-repos/serena` (cloned 2026-07-08). Audience: a power user who builds MCP endpoints and wants transferable patterns.

The one-line thesis: Serena treats the codebase as a symbol tree served by a Language Server (LSP), not as text. Every read tool returns symbols (name-path + location + optional body), and every edit tool operates on symbol boundaries the LSP computes — so the model never has to quote large spans or count lines. Everything else (tool registry, memory graph, context/mode layering, progressive-shortening) is scaffolding around that core bet.

0. Repo shape (orientation)

  • src/serena/ — the agent, MCP server, tools, config, prompts, memory.
    • tools/ — the MCP tool surface, one file per concern (symbol_tools.py, file_tools.py, memory_tools.py, workflow_tools.py, config_tools.py, cmd_tools.py, query_project_tools.py, jetbrains_tools.py), plus tools_base.py (the framework).
    • symbol.py — the semantic-retrieval engine (symbol model, name-path matcher, retriever).
    • code_editor.py — symbol-boundary editing.
    • memories/memory_manager.py — the project-memory store.
    • resources/config/ — contexts/, modes/, prompt_templates/ (the injected prompts, as data not code).
  • src/solidlsp/ — a vendored, language-agnostic LSP client ("SolidLSP") with 70 language-server integrations under language_servers/. This is the moat: it normalizes 70 LSPs into one symbol API.
  • src/interprompt/ — a small multilingual Jinja prompt-template engine.

1. MCP tool surface — design principles

1.1 Tools are classes; the docstring is the tool description; the apply signature is the schema

src/serena/tools/tools_base.py:142 — class Tool(Component). The pattern (documented in the class comment at line 143):

  • Each tool subclasses Tool and implements apply(self, ...) -> str.
  • The class docstring becomes the high-level tool description; the apply docstring + type hints are fed to MCP via func_metadata(apply_fn, skip_names=["self","cls","session_id"]) (tools_base.py:267) to auto-generate the JSON schema and per-parameter descriptions.
  • Tool name is derived from the class name by stripping a Tool suffix and converting to snake_case (get_name_from_cls, tools_base.py:193): GetSymbolsOverviewTool → get_symbols_overview.

Why it matters: there is no hand-maintained schema. The thing the LLM reads (docstring) and the thing that validates the call (signature) are the same object, so they can never drift. If you build MCP endpoints in Python, this "docstring-as-contract" is the single highest-leverage steal.

1.2 Capability is expressed with marker mixins, not config flags

tools_base.py:82-119 defines empty marker classes:

class ToolMarkerCanEdit(ToolMarker): ...
class ToolMarkerDoesNotRequireActiveProject(ToolMarker): ...
class ToolMarkerOptional(ToolMarker): ...          # disabled by default
class ToolMarkerSymbolicRead(ToolMarker): ...
class ToolMarkerSymbolicEdit(ToolMarkerCanEdit): ...
class ToolMarkerBeta(ToolMarker): ...

A tool declares class ReplaceSymbolBodyTool(EditingToolWithDiagnostics) or class FindSymbolTool(Tool, ToolMarkerSymbolicRead). The framework then reasons over capabilities generically: can_edit() is just issubclass(cls, ToolMarkerCanEdit) (tools_base.py:217); read-only mode is enforced by filtering out can_edit() tools; the system prompt is templated on 'ToolMarkerSymbolicRead' in available_markers (see §5). Capability travels with the type — you can't forget to register it.

1.3 Auto-registering singleton registry with a tombstone list

tools_base.py:558 — @singleton class ToolRegistry. On construction it walks iter_subclasses(Tool) and registers every concrete subclass (inclusion_predicate = lambda c: "apply" in c.__dict__, i.e. only classes that actually implement apply). Zero manual registration.

Notable detail (tools_base.py:560): a hardcoded _deleted_tools list of retired tool names (think_about_collected_information, summarize_changes, check_onboarding_performed, …). check_valid_tool_name warns-and-skips deleted names instead of raising (:659). This is how you evolve a public tool surface without breaking configs that still reference old names — keep tombstones.

1.4 One executor path with retries, timeouts, session-awareness, and self-healing LSP restart

apply_ex (tools_base.py:330) is the central invocation wrapper every tool goes through:

  • Derives a stable session_id from id(mcp_ctx.session) and injects it only into tools whose apply declares a session_id param (_is_session_aware, :166) — session-awareness is opt-in per tool, invisible to MCP schema (session_id is in skip_names).
  • Guards: refuses if the tool is inactive (:358) or if a project is required but none is active — and the error message lists the known projects so the model can self-correct (:369).
  • Self-healing: if a call raises SolidLSPException whose language server has terminated, it restarts that language server and retries the call once (:382-397). Crash recovery is invisible to the model.
  • Runs the whole thing on the agent's task executor with tool_timeout (:424), so a hung LSP can't wedge the server.
  • Client-quirk sanitization: _sanitize_input_param undoes &lt;/&gt; HTML-escaping that some clients apply to </> in regex args (:180).

Steal: a single choke-point wrapper around tool execution (guards → self-heal → timeout → usage logging) beats scattering try/except into each tool.

1.5 Progressive-shortening: results degrade gracefully instead of truncating

_limit_length(result, max_answer_chars, shortened_result_factories) (tools_base.py:281). Every heavy tool passes a list of closures producing progressively shorter renderings, tried in order until one fits the char budget. Examples:

  • FindReferencingSymbolsTool (symbol_tools.py:318-330): full refs-with-code → refs-without-surrounding-lines → per-file counts → bare count.
  • SearchForPatternTool (file_tools.py:625-638): matches-with-context → match-lines-only → per-file counts → "found N matches in M files."
  • GetSymbolsOverviewTool (symbol_tools.py:76-88): full → depth-0 overview → symbol-counts-by-kind.

Instead of blindly cutting bytes, the tool answers a coarser question that still fits. This is a genuinely novel MCP pattern most builders miss.

1.6 The full default tool surface (what's exposed, and the deliberate omissions)

Grouped by module (from the source):

  • Symbol read (symbol_tools.py): get_symbols_overview, find_symbol, find_referencing_symbols, find_implementations (optional), find_declaration, get_diagnostics_for_file, get_diagnostics_for_symbol (optional).
  • Symbol edit: replace_symbol_body, insert_after_symbol, insert_before_symbol, rename_symbol, safe_delete_symbol.
  • File (file_tools.py): read_file, create_text_file, list_dir, find_file, replace_content, replace_in_files, and optional line-level ops (delete_lines, replace_lines, insert_at_line).
  • Memory (memory_tools.py): write_memory, read_memory, list_memories, delete_memory, rename_memory, edit_memory.
  • Workflow (workflow_tools.py): onboarding, initial_instructions, serena_info.

Two design decisions worth copying:

  1. Line-level edit tools are ToolMarkerOptional (disabled by default) (file_tools.py:467,493,527). Serena wants the model to edit at the symbol level; raw line edits are a fallback you must opt into. The default surface nudges the model toward the safer, more robust primitive.
  2. find_referencing_symbols never includes the referencing symbol's body (symbol_tools.py:277, comment: "It is probably never a good idea…") — but does attach a 3-line snippet around each reference (:297). The tool's granularity is tuned to what's actually useful, not what's mechanically possible.

2. Semantic retrieval — how it beats grep

This is the heart of the repo: src/serena/symbol.py.

2.1 The "name path": a language-agnostic symbol address

class NamePathMatcher (symbol.py:142) and get_name_path (symbol.py:346). A symbol is addressed by its path in the in-file symbol tree: the method my_method in class MyClass is MyClass/my_method. Rules (from the docstring, mirrored in tool docs):

  • simple name method → matches any symbol with that name;
  • relative path class/method → matches any symbol whose name-path ends with that suffix;
  • absolute /class/method → exact full-path match within the file;
  • overloads disambiguated with an index: MyClass/my_method[1].

Matching is implemented by walking the symbol's ancestors in reverse and comparing suffix-first (matches_reversed_components, symbol.py:200), with substring matching allowed only on the last (leaf) component. This gives the model a stable, human-readable handle for any symbol that works identically across all 70 languages — no line numbers, no byte offsets.

2.2 Retrieval is LSP calls, wrapped

class LanguageServerSymbolRetriever (symbol.py:578) is a thin semantic layer over SolidLSP:

  • find(name_path_pattern, ...) → lang_server.request_full_symbol_tree(...) then filters the tree with NamePathMatcher (symbol.py:734). If a specific file is given it picks the best LSP for that file type; otherwise it iterates all active language servers.
  • find_referencing_symbols → LSP textDocument/references, then maps each reference back to its containing symbol (symbol.py:809) — so "who calls this?" returns symbols, not raw line hits.
  • find_implementing_symbols → LSP textDocument/implementation.
  • find_declaration → LSP go-to-definition from a cursor position.
  • request_info_for_symbol → LSP textDocument/hover (docstring + signature), sanitized across the several hover response shapes (symbol.py:586-617).

The discriminating difference vs grep: references resolve through the type system, so process the method is never confused with process the variable, and results are attributed to the enclosing function/class.

2.3 Two cost-control mechanisms a power user should copy

  1. Hover time-budget (request_info_for_symbol_batch, symbol.py:631). Hover is expensive, so batch lookups are grouped by file (open each file once), and a wall-clock budget (symbol_info_budget, default 5s) is checked before each hover; once exceeded, remaining symbols get info=None. Partial results beat timeouts. This is a general RAG-over-LSP pattern: bound the expensive enrichment, degrade to nulls.
  2. "Low-level" filtering for overviews (is_low_level, symbol.py:244: symbol_kind >= SymbolKind.Variable). Overviews and diagnostics-ownership deliberately drop variables/constants and climb to the structural container (_normalize_symbol_for_diagnostics, symbol.py:1000), because a high-level map shouldn't be polluted with locals.

2.4 Compact, tunable serialization

LanguageServerSymbol.to_dict (symbol.py:449) has ~a dozen boolean knobs (name_path, name, kind, location, body, body_location, depth, children_*, child_inclusion_predicate). Each tool renders exactly the fields it needs at exactly the depth it needs — the same symbol object serves an overview (names+kinds, no bodies) and an edit-prep read (one body, no children).

On top of that, SymbolDictGrouper (symbol.py:1228) regroups the flat dict list by keys like kind or relative_path and collapses singletons ({"name":"foo"} → "foo"; {"name":"foo","children":{...}} → {"foo":{...}}, symbol.py:1294) to shave tokens. Cleverly, the grouper validates its group-keys against the TypedDict annotations at import time (symbol.py:1252) — a config typo fails at startup, not mid-conversation.

2.5 The default read protocol (encoded in the prompt)

The system prompt (system_prompt.yml:22-31) teaches the loop directly: get_symbols_overview on a file → find_symbol with include_body=False, depth=1 to list a class's methods → find_symbol include_body=True for only the bodies you need → find_referencing_symbols for relationships. "You only read the bodies of symbols when you need to." The tool granularity and the prompt are co-designed to produce token-minimal exploration.

3. Editing — precise, boundary-aware, safe

3.1 Symbol-boundary edits with newline hygiene

src/serena/code_editor.py. replace_body (code_editor.py:103) looks up the symbol's start/end position from the LSP, strips the new body of surrounding whitespace, deletes the old span, inserts the new — all inside an edited_file_context that writes back on clean exit (:76). The interesting engineering is in insert ops: insert_after_symbol/insert_before_symbol (:139, :180) compute how many blank lines a definition of that kind should be separated by (is_neighbouring_definition_separated_by_empty_line, symbol.py:255 — true for functions/classes/methods) and normalize leading/trailing newlines to match. The model supplies content; the tool owns placement and spacing. That is why symbol edits are more robust than the model hand-counting whitespace.

3.2 rename_symbol uses the LSP's own refactoring

LanguageServerCodeEditor.rename_symbol (code_editor.py:356) calls LSP textDocument/rename, gets back a WorkspaceEdit, and applies it — including cross-file TextEdits and file-rename operations (_workspace_edit_to_edit_operations, :323). This is real, type-aware, project-wide renaming, not find-replace.

3.3 safe_delete_symbol refuses to delete referenced code

SafeDeleteSymbol.apply (symbol_tools.py:683): first requests references; if any exist, it returns the reference locations instead of deleting (:718). Only truly-unreferenced symbols are removed. A destructive op made safe by using the same semantic index that powers reads.

3.4 Multi-file replace with a dry-run / occurrence-id protocol

ReplaceInFilesTool (file_tools.py:234) is the most sophisticated editing tool and the strongest steal for anyone building a bulk-edit MCP:

  • Dry-run first: returns every prospective change as a minimal diff, each tagged with an occurrence_id of the form <path>:<index>@<digest>.
  • Apply by id: caller passes back the ids to apply; if any id is stale (file changed since the dry run) nothing is applied and the model is told exactly why per id (_resolve_occurrence_ids, :407).
  • Guards for blind applies: expected_count (mismatch → abort + show diffs), and ambiguity detection when the pattern matches again inside its own match (over-matching → abort, :326).
  • TOCTOU re-validation at write time: if the editor's content differs from what was scanned, it re-derives occurrences from the authoritative content and re-checks ids before writing (_apply_occurrences, :450).

The whole tool is engineered so a wrong regex costs one call and returns a selectable list, never a silent bad edit. The docstring actively coaches the model toward regex-with-wildcards (file_tools.py:187: "you cannot make mistakes, because if the regex matches multiple occurrences… an error will be returned").

3.5 Diagnostics-aware editing (built, currently gated off)

EditingToolWithDiagnostics (tools_base.py:442) can snapshot LSP diagnostics before an edit and diff them after, appending a diagnostics[warning-or-higher] block to the result. It's globally disabled (ENABLE_DIAGNOSTICS = False) with an honest comment (:447): per-edit diagnostics are noisy because intermediate edits intentionally introduce transient errors resolved by later edits. Worth noting as a considered non-feature.

4. Memory / onboarding — a referenced graph, not a dump

4.1 Memories are markdown files with a mem: reference convention

src/serena/memories/memory_manager.py. Memories are .md files under .serena/memories/ (project) or a global dir. Names can nest with / into topic folders. The tool surface (memory_tools.py) is deliberately small: write / read / list / delete / rename / edit. Two details do a lot of work:

  • Cross-memory references use a mem: prefix inside backticks (e.g. `mem:frontend/core`). rename_memory rewrites every mem:oldname reference across all memories (rename_memory_and_propagate_references, memory_manager.py:331) using a boundary-anchored regex so it can't match inside a longer name (:123). Memories form a maintained link graph.
  • LLM-mistake tolerance: _sanitize_name (:73) strips a stray mem: prefix, a .md suffix, and normalizes separators — the model's sloppy names just work. .. path segments are rejected for security (:153).

4.2 The discovery model: progressive graph traversal from mem:core

The shipped memory_maintenance.md template (resources/memory_maintenance.md) states the doctrine plainly:

  • Agents are given only the list of memory names at startup; they infer relevance from names and read on demand.
  • mem:core is the graph root → references domain memories → which reference more specific ones. Depth scales with project complexity.
  • "Memories themselves should not contain information about when to read them; this is the responsibility of the referring memory." The pointer carries the "when," the target carries the "what." This is the key idea and it's the opposite of most RAG chunking.
  • Style: "Dense agent notes, not prose docs… invariants, terse bullets… durable and generalizable, not task-local," with an explicit add/update threshold: only stable, non-obvious conventions; never one-off task notes or generic framework knowledge.

4.3 Onboarding is a tool that returns instructions to write memory

OnboardingTool (workflow_tools.py:10) doesn't scan the repo itself — it returns a prompt (from simple_tool_outputs.yml:5, the onboarding_prompt template) that tells the model to:

  1. First read mem:memory_maintenance (the conventions) — seeded from the shipped template on first use if absent (ensure_memory_maintenance_memory, memory_manager.py:89, with a global-overrides-project precedence chain).
  2. Then produce a fixed target layout of memories: core, tech_stack, suggested_commands, conventions, task_completion (per-module <module>/core for split repos).
  3. Acquire info with the tools, "read only the files needed; do not load entire directory trees."
  4. Hard requirement (simple_tool_outputs.yml:40): "the onboarding is only complete once you have actually called write_memory for each memory above — do not summarize the content in chat and skip writing."

So "onboarding" is a self-executing instruction that turns the first session's exploration into durable artifacts every future session reads by name. Several "thinking" tools (check_onboarding_performed, summarize_changes) were later deleted (tools_base.py:560) — the team trimmed prompt-nudge tools over time.

5. Prompts / context — layered, data-driven, capability-templated

5.1 Three-layer prompt assembly: base system prompt + one context + N modes

Prompts live as YAML data, not code, under resources/config/:

  • prompt_templates/system_prompt.yml — the base "Serena Instructions Manual."
  • contexts/*.yml — where Serena runs (claude-code, codex, vscode, desktop-app, ide, chatgpt, … 15 of them). A context sets excluded_tools, single_project, structured_tool_output, and a prompt fragment.
  • modes/*.yml — how it's operating (planning, editing, interactive, one-shot, onboarding, no-onboarding, no-memories, …). Modes mostly toggle tool availability.

The base prompt (system_prompt.yml:52-58) interpolates context_system_prompt and loops mode_system_prompts. SerenaConfig (config/serena_config.py) composes them: base_modes = ("interactive","editing") plus project added_modes, minus each layer's excluded_tools. planning.yml is the cleanest example — it's almost entirely a list of edit tools to exclude (modes/planning.yml:5-14), producing a read-only agent from the same toolset.

5.2 Jinja-templated on runtime capabilities

The system prompt is a Jinja template keyed on the actual live tool/marker set: {% if 'ToolMarkerSymbolicRead' in available_markers %}, {% if 'search_for_pattern' in available_tools %}, {{ tool_names['find_symbol'] }} (system_prompt.yml:14-31). If a context excludes symbolic tools, the paragraphs teaching them vanish. The prompt describes only the tools that are present — no dangling references to unavailable tools, and tool names are injected (not hardcoded) so renaming a tool can't desync the prompt.

5.3 Context-specific override that fights the host agent's defaults

The most instructive prompt is claude-code.yml (context) and system_prompt.yml:62 (cc_system_prompt_override). Serena knows Claude Code has strong built-in Read/Grep/Edit habits, so the context prompt is an explicit, almost adversarial override:

Read  -> FORBIDDEN for discovery. Use get_symbols_overview, then find_symbol...
Edit  -> FORBIDDEN. Use replace_symbol_body / insert_*_symbol / replace_content.
Disallowed reasoning. Do NOT use any of the following to justify Read/Edit...:
  - "I already know the path"
  - "one Read call is faster than three Serena calls"

It even names the rationalizations the model tends to use and pre-empts them, and includes a task→tool mapping table (system_prompt.yml:82-97). Lesson for MCP authors: when your tools compete with a host agent's native tools, you must actively redirect, list the excuses, and give a lookup table — a polite "prefer our tools" is ignored.

5.4 The "call this tool first" bootstrap for clients that ignore system prompts

Many MCP clients won't let a server set the real system prompt. Serena's workaround (workflow_tools.py:32, InitialInstructionsTool; system_prompt.yml:5, connection_prompt): expose an initial_instructions tool whose docstring says "call this immediately after you are given your task," which returns the full manual as a normal message. The system prompt is delivered as a tool result. InitialInstructionsTool is also marked ToolMarkerDoesNotRequireActiveProject so it works before any project is loaded.

6. Language-agnosticism — the underrated moat

src/solidlsp/ is a standalone, vendored LSP client with 70 language-server integrations (ls -1 src/solidlsp/language_servers/ | wc -l → 70). Language is a str-Enum (solidlsp/ls_config.py:51) and per-language behavior is dispatched with match self: blocks (:300,316,547) — e.g. picking a language server, file extensions, ignore rules. The entire serena/ layer speaks only the normalized SolidLanguageServer + UnifiedSymbolInformation API; adding a language means adding a language_servers/*.py and enum entry, not touching any tool. The retriever even picks the best LSP per file so a polyglot repo (PHP served by the PHP LSP, TS by tsserver) works in one session (symbol.py:747-756). The 70-language surface is what turns "a Python code tool" into "a code tool."

7. Config surface (for the MCP builder)

From config/serena_config.py: per-tool controls are excluded_tools, included_optional_tools, fixed_tools (:150-158); modes via base_modes/added_modes/default_modes (:181-194); safety/tuning via read_only, symbol_info_budget, default_max_tool_answer_chars, tool_timeout, ignored_paths, ignore_all_files_in_gitignore, read_only_memory_patterns, ignored_memory_patterns, language_backend (LSP vs JetBrains). Templates ship as *.template.yml alongside the resources. The design lets an operator produce a minimal tool surface per deployment (a context with single_project: true even drops project-switching tools).

TOP STEALS (highest-ticket, ranked)

  1. Docstring-as-contract tools. One Python method's docstring + type hints generate the MCP description and the validating schema (tools_base.py:249-267, :220). Name derived from class name. Impossible for the model-facing description and the validation to drift. — the single best pattern for anyone building MCP tools in Python.

  2. Progressive-shortening result rendering. Pass a tool a list of closures that answer the same question at coarser resolution; return the first that fits the char budget (tools_base.py:281; used in symbol_tools.py:318, file_tools.py:625). Degrade to "per-file counts" or "N matches," never blind truncation.

  3. Symbol-address ("name path") + symbol-boundary edits. Address code as Class/method (symbol.py:142, :346) and edit by replacing the LSP-computed span with the tool owning newline/spacing hygiene (code_editor.py:103-206). The model never quotes big spans or counts lines — the durable win over line/diff-based editing.

  4. Dry-run + occurrence-id + staleness-checked bulk edit. ReplaceInFilesTool (file_tools.py:234): preview → apply-by-id → abort-and-explain on any stale/ambiguous/over-matching case, with TOCTOU re-validation at write time (:407, :450). A wrong regex costs one call and returns a selectable list, never a silent bad edit.

  5. Referenced memory graph + onboarding-as-instruction. Memories are dense markdown addressed by name, linked via mem: refs that survive renames (memory_manager.py:331); the pointer holds the "when to read," the target holds the "what" (resources/memory_maintenance.md). Onboarding is a tool that returns a prompt forcing the first session to persist a fixed memory layout via write_memory (workflow_tools.py:10, simple_tool_outputs.yml:5).

(Runner-up, worth a mention: the capability-templated, three-layer prompt system — base + context + modes, Jinja-gated on the live tool/marker set, with an adversarial host-tool override that names the model's own rationalizations — system_prompt.yml, contexts/claude-code.yml.)

All citations verified against the cloned source at ~/Developer/reference-repos/serena on 2026-07-08. No paths inferred. The only non-code inference, marked here: the "70 language servers" figure is a directory file count, and ENABLE_DIAGNOSTICS=False means the diagnostics-diff feature in §3.5 ships disabled.