Tutorial: Code Optimization ============================== This tutorial walks a beginner end-to-end through running **FermiLink**'s benchmark-gated code optimizer on a scientific package (Python, C, C++, or Fortran). It is **self-contained**: you can follow it from a fresh shell without reading other pages. ``fermilink optimize`` iteratively proposes source-level changes to a target package, benchmarks each candidate, validates correctness, and **accepts only changes that improve measured performance while preserving results**. You describe your intent in a short structured markdown file (``goal.md``); **FermiLink** will perform the optimization and return a branch with the accepted commits. For general-purpose HPC simulation workflows (``exec``, ``loop``, ``research``, ``reproduce``), see :doc:`tutorial_hpc`. Prerequisites ~~~~~~~~~~~~~ You need the following available locally (all can be user-local, no sudo required): - Python ``>= 3.11`` - ``git`` on ``PATH`` (optimize campaigns run inside a git worktree) - Node.js + ``npm`` (for the Codex agent CLI; other provider CLIs may use their own installers) - A C / C++ / Fortran toolchain **if** the package you want to optimize is compiled (e.g. ``gcc``, ``g++``, ``gfortran``, ``cmake``, ``make``) This tutorial assumes you will use the OpenAI **Codex** CLI as the agent backend, since it currently gives the strongest optimization results with ``gpt-5.4`` at ``xhigh`` reasoning effort. Step 1. Install the Codex CLI and authenticate ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Install the Codex CLI globally (or user-locally via ``npm --prefix``) and log in once: .. code-block:: bash npm i -g @openai/codex codex login After ``codex login`` completes you should be able to run ``codex`` interactively without being prompted for credentials. .. note:: **FermiLink** also supports Claude, Gemini, and OpenCode agents. For optimization we recommend Codex because of its superior code-editing and profiling behavior at high reasoning effort as well as its relative affordability. If you prefer a different provider, see :doc:`choosing_agent`. Step 2. Install **FermiLink** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Install **FermiLink** into the Python environment you want to drive the optimizer from. Note that the Python launcher in Step 4 creates a **separate** per-campaign venv for the target package, so this environment only needs **FermiLink** itself. .. code-block:: bash pip install fermilink This installs the ``fermilink`` CLI and the two opinionated optimize launchers (``fermilink-optimize-python`` and ``fermilink-optimize-cpp``) onto your ``PATH``. Step 3. Set the agent runtime policy ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Optimization agents need to compile code, run benchmarks, and profile hot paths, so the default restricted sandbox is too tight. Switch the Codex agent to bypass-sandbox mode with the most capable model and highest reasoning effort: .. code-block:: bash fermilink agent codex --bypass-sandbox \ --model gpt-5.4 \ --reasoning-effort xhigh # OpenCode is also supported when you prefer OpenCode-managed profiles fermilink agent opencode --bypass-sandbox \ --model openai/gpt-5.5 \ --reasoning-effort xhigh # verify what is active fermilink agent --json This mirrors the recommended HPC configuration in :doc:`tutorial_hpc`. ``xhigh`` reasoning effort is important for optimization: the agent needs deep deliberation to reason about correctness constraints while restructuring hot paths. .. warning:: Bypassing the sandbox lets the agent execute arbitrary shell commands in your user account. **Never run this as root.** Use a dedicated non-root account and keep regular backups. Also run campaigns on a **clean clone** of the target package so the original checkout stays untouched (the launchers in Step 4 enforce this via a sibling git worktree). Step 4. Write a ``goal.md`` file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``goal.md`` file is the only input you author by hand. It is a short structured markdown file describing **what** to optimize and **what invariants must hold**. **FermiLink** converts it into a deterministic benchmark contract automatically. For the full authoring guide, see :ref:`Goal files for code optimization `. Also view the `example goal files at Project Optimization `_ for inspiration -- these are all real goals that have driven successful campaigns on open-source scientific packages. A minimal goal file looks like this: .. code-block:: markdown # Optimization Goal ## Package pyscf ## Language python ## Target Optimize DIIS behavior for SCF convergence in PySCF, focusing on reduced overhead and faster convergence paths. ## Editable Scope - pyscf/lib/diis.py - pyscf/scf/** ## Performance Metric Minimize end-to-end SCF kernel time from `mf.kernel()`. ## Correctness Constraints - Total SCF energy absolute delta <= 5e-8 Hartree vs baseline - All benchmark cases must converge within baseline cycle limits ## Representative Workloads - train-rhf-benzene: benzene / RHF / 6-31g** / diis_space=12 - train-uhf-allyl: allyl radical / UHF / spin=1 / def2-TZVP - test-rhf-c3h7oh: C3H7OH / RHF / 6-31g** ## Build ```bash pip install -e . ``` ## Notes Keep benchmark behavior deterministic with pinned thread counts. Save this file anywhere -- e.g. ``~/goals/pyscf-diis.md``. .. tip:: If you are new to writing optimization goals, ask **FermiLink** itself. From any directory, run ``fermilink init`` then ``codex`` (or ``claude``, ``gemini``, or ``opencode``) and tell the agent: *"Use the optimize-goal-authoring skill to write a well-structured optimization goal for in ."* See :ref:`Goal files for code optimization ` for the full prompt. Step 5. Launch your first campaign ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The easiest way to start a campaign is the shipped launcher matching your language. The launchers handle all the plumbing, including creating a sibling git worktree and excluding ``fermilink optimize`` runtime data from git tracking. **Python packages** (PySCF, ASE, PyTorch-based solvers, ...): .. code-block:: bash cd /path/to/pyscf # your clean clone of the target package fermilink-optimize-python ~/goals/pyscf-diis.md --branch fermilink-optimize/pyscf-diis **C / C++ / Fortran packages** (LAMMPS, Quantum ESPRESSO, GROMACS, ...): .. code-block:: bash cd /path/to/lammps fermilink-optimize-cpp ~/goals/lammps-tip4p.md --branch fermilink-optimize/lammps-tip4p That is all. The launcher: 1. Creates a sibling git worktree next to the source repo so your original checkout stays untouched. 2. Names the worktree branch ``fermilink-optimize/-`` (derived from the goal file name). 3. (Python launcher only) Creates a per-branch venv under ``/venvs/`` and ``pip install fermilink`` into it. 4. Calls ``fermilink optimize `` inside the worktree. Progress streams live to your terminal. You can check the current optimization status at any time from another shell: .. code-block:: bash fermilink optimize status .. note:: If you are not already inside a git repo, pass ``--project-root /path/to/clean/clone`` instead of ``cd``\ ing first. See the :ref:`detailed launcher options ` section below. Step 6. What happens during a run ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Understanding the two internal stages helps you read the log stream and intervene if something looks off. **Goal-mode pipeline (one-time setup)** When you submit a ``goal.md`` for the first time, **FermiLink** runs a two-phase generation pipeline **before** starting the optimization loop: 1. **Source analysis**: an agent reads your target source code and produces a structured JSON analysis of the package and correctness boundaries. 2. **Benchmark generation**: a second agent writes a ``benchmark.yaml`` deterministic contract and a ``benchmark_runner.py`` driver script, which are validated and placed under ``.fermilink-optimize/autogen/``. A copy of your input ``goal.md`` is preserved alongside them. Once generation is done the campaign switches to the standard optimization loop. **Campaign lifecycle (repeated until convergence or iteration cap)** 1. **Baseline**: the generated benchmark suite runs on the unmodified source to establish incumbent performance. 2. **Worker turn**: an AI worker agent proposes a single candidate source-level change inside an isolated git worktree. 3. **Benchmark**: the candidate is benchmarked. 4. **Controller turn**: a separate controller agent evaluates correctness (against your ``## Correctness Constraints``) and performance, then **accepts or rejects** the candidate. 5. **Iterate**: repeat from step 2 until the iteration cap, the consecutive-rejection limit, or ``--forever`` termination. Runtime data for ``fermilink optimize`` is persisted under ``.fermilink-optimize/``. Step 7. Resume from an interrupted run ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Long campaigns occasionally get interrupted. You can resume exactly where you left off by passing ``--resume`` through to the launcher: .. code-block:: bash # Python fermilink-optimize-python --goal ~/goals/pyscf-diis.md --branch fermilink-optimize/pyscf-diis \ -- --resume # C/C++/Fortran fermilink-optimize-cpp --goal ~/goals/lammps-tip4p.md --branch fermilink-optimize/lammps-tip4p \ -- --resume Anything after ``--`` is forwarded verbatim to the underlying ``fermilink optimize`` call, so ``-- --resume`` means "add ``--resume`` to ``fermilink optimize``". What ``--resume`` does: - Reuses the existing sibling worktree and branch (does not recreate them). - Reuses the already-generated ``benchmark.yaml`` and runner under ``.fermilink-optimize/autogen/`` -- the expensive two-phase goal pipeline is **skipped**. - Reloads controller and worker memories and continues the iteration loop from the last persisted state. - Preserves accepted commits already in the branch history. You can combine ``--resume`` with other flags, e.g. extending the iteration cap or the per-run benchmark timeout: .. code-block:: bash fermilink-optimize-cpp --goal ~/goals/lammps-tip4p.md --branch fermilink-optimize/lammps-tip4p \ -- --resume --max-iterations 60 --timeout-seconds 6000 Step 8. Power user: tune the generated ``benchmark.yaml`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After the first goal-mode run, **FermiLink** has written a fully deterministic benchmark contract under: .. code-block:: text .fermilink-optimize/autogen/ ├── goal.md # copy of your input ├── benchmark.yaml # generated deterministic contract └── benchmark_runner.py # generated driver script This ``benchmark.yaml`` is what actually drives the optimization loop. For more deterministic behavior, **FermiLink does not re-read ``goal.md`` during iterations**. That means once you are happy with the auto-generated contract (or want to hand-tune it), you can skip goal mode entirely and drive the optimizer directly in **expert mode**: .. code-block:: bash fermilink optimize \ --benchmark .fermilink-optimize/autogen/benchmark.yaml \ --resume Why you might want to tune the YAML by hand: - **Target section**: add or remove editable file globs to narrow the agent's scope. - **Correctness tolerances**: tighten or loosen numerical tolerances in ``field_tolerances`` based on what you observed in the baseline run. - **Runtime hooks**: edit ``runtime.pre_commands`` to add a custom rebuild recipe of the package. - **Per-run timeout**: change ``runtime.timeout_seconds`` if some cases are outliers. The rule of thumb: use ``goal.md`` to bootstrap the contract, then treat ``benchmark.yaml`` as the source of truth for further iterations. Step 9. Check campaign progress ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ At any time, inside the modified repo, inspect the campaign state: .. code-block:: bash fermilink optimize status Troubleshooting quick checks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - **Dirty working tree refused**: commit or stash your changes, or pass ``--allow-dirty-base`` to the launcher if you know what you are doing. - **Every candidate rejected on correctness**: your ``## Correctness Constraints`` are likely too tight. See :doc:`writing_goal_md` for physically meaningful tolerance guidance. - **Benchmark always times out**: bump ``--timeout-seconds``, or pick smaller ``train-`` workloads. The optimizer never sees the ``test-`` cases during iteration. - **Resume does nothing**: make sure you are in the same worktree (the sibling one created by the launcher), not the original source checkout. ---- Detailed reference: launcher and ``fermilink optimize`` options --------------------------------------------------------------- The Step 5 and Step 7 commands above cover 95% of day-to-day usage. The remaining sections document every flag for the moments you need fine-grained control. .. _optimize-launcher-options: Common launcher options ~~~~~~~~~~~~~~~~~~~~~~~ Shared between ``fermilink-optimize-python`` and ``fermilink-optimize-cpp``. .. list-table:: :header-rows: 1 :widths: 35 65 * - Flag - Description * - ``--goal PATH`` - Goal markdown file (**required**). * - ``--project-root PATH`` / ``--repo PATH`` - Clean git repo to optimize. Defaults to the current working git repo root. * - ``--branch NAME`` - Worktree branch name. Default ``fermilink-optimize/-``. * - ``--base-ref REF`` - Base ref used when creating a new branch. Defaults to ``origin/HEAD`` when available, otherwise the current branch or ``HEAD``. * - ``--worktree-root PATH`` - Parent dir for generated worktrees. Default ````. * - ``--worktree-name NAME`` - Explicit worktree directory name. Default ``-``. * - ``--hpc-profile PATH`` - HPC profile JSON forwarded to ``fermilink optimize`` for SLURM submission. See :doc:`tutorial_hpc`. * - ``--worker-provider NAME`` - Override the worker-agent provider (e.g. ``claude``, ``gemini``, ``opencode``). * - ``--worker-model MODEL`` - Override the worker-agent model. Inherits sandbox/reasoning from the active agent policy. * - ``--isolate-fermilink-home`` - Run with a campaign-local ``FERMILINK_HOME`` so settings do not bleed between campaigns. * - ``--fermilink-home PATH`` - Explicit ``FERMILINK_HOME`` path (implies isolation). * - ``--fermilink-bin BIN`` - Which ``fermilink`` executable to run. Default ``fermilink``. * - ``--allow-dirty-base`` - Allow uncommitted changes in ``--project-root``. * - ``--dry-run`` - Print the resolved ``fermilink optimize`` command and exit. * - ``-h`` / ``--help`` - Show the launcher help. Python-launcher-only options ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``fermilink-optimize-python`` additionally provisions a per-branch virtualenv with ``pip install fermilink`` inside it: .. list-table:: :header-rows: 1 :widths: 35 65 * - Flag - Description * - ``--python-bin BIN`` - Python executable used to create the venv. Default ``python3``. * - ``--venv-root PATH`` - Parent dir for per-branch venvs. Default ``/venvs``. * - ``--venv-name NAME`` - Override the default venv directory name. * - ``--venv-path PATH`` - Explicit venv path (overrides ``--venv-root``/``--venv-name``). Extra optimize arguments (after ``--``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Everything after ``--`` on the launcher command line is forwarded verbatim to ``fermilink optimize``. These are the flags you most often want to pass through. .. list-table:: :header-rows: 1 :widths: 40 60 * - Flag - Description * - ``--baseline-only`` - Run only the baseline benchmark and exit. Useful for sanity checks. * - ``--plan-only`` - Initialize state and validate inputs without running the loop. * - ``--resume`` - Resume an existing campaign from local state (see Step 7). * - ``--max-iterations `` - Override the campaign iteration cap. * - ``--worker-max-iterations `` - Override the per-turn worker iteration cap. * - ``--stop-on-consecutive-rejections `` - Override the rejection-based early-stop threshold. * - ``--timeout-seconds `` - Override the per-run benchmark timeout. * - ``--hpc-profile `` - HPC profile for SLURM job submission. * - ``--forever`` - Run indefinitely until interrupted. * - ``--allow-dirty`` - Allow startup from a dirty git working tree. * - ``--sandbox `` - Provider sandbox override for optimize agent turns. Expert mode: run directly from ``benchmark.yaml`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Once you have a ``benchmark.yaml`` (either hand-written or generated by goal mode as shown in Step 8), you can skip the launchers and the goal pipeline entirely: .. code-block:: bash fermilink optimize \ --benchmark \ [--branch NAME] \ [--resume] \ [--max-iterations N] \ [--timeout-seconds N] \ [--hpc-profile PATH] Further reading --------------- - :doc:`writing_goal_md` -- full authoring guide for ``goal.md`` files, including every required section, tolerance conventions, and domain-specific examples. - :doc:`tutorial_hpc` -- how to wire ``--hpc-profile`` for SLURM so optimize benchmarks run on a cluster instead of your laptop. - :doc:`choosing_agent` -- picking between Codex, Claude, Gemini, and OpenCode as the worker or controller agent. - :doc:`configuration` -- ``FERMILINK_HOME`` and related environment variables. - :doc:`usage_advanced_configuration` -- reusable pipelines and memory-driven skill updates.