Methodology
The observatory evaluates how well spec-driven development approaches work in practice by watching real projects over time — not by quoting marketing pages. Everything on the site comes from one of two places: an automated metrics pipeline that humans never edit, or a curated editorial assessment. Assessments may be researched and drafted with AI assistance, but they are published through the same reviewable git workflow as any other content change. This page explains what we track, where each number comes from, and how to read the ratings.
What we track
Frameworks
Documented methodologies for driving AI-assisted development from specifications. Each page records the framework's approach, workflow, supported tools, maturity, strengths, and limitations.
Projects
Public repositories that use one of those frameworks. Each page tracks how the specs are structured, how actively the code is developed, whether specs and code stay in sync, and how maintenance plays out over time.
How projects enter the observatory
Tracked projects come from two sources: submissions, and periodic discovery sweeps
that search GitHub for each framework's fingerprint — the files its tooling scaffolds into a user's repo
(an openspec/ directory, a .kiro/specs/ tree, and so on). Sweep results pass
through a fixed funnel before anything is published:
- Mechanical filters. A script drops forks, archived repos, the frameworks' own repos and official demos, templates and starters, repos with no pushes in the last 90 days, and repos whose spec directory holds specs for fewer than two changes or features.
- Star floor. Candidates need at least 500 GitHub stars to be considered for a project page. The floor is a triage heuristic that keeps review effort on projects with real adoption — not a quality judgment. When a framework has no candidate above the floor, a maintainer may still track its best real-world usage, and the project page says so.
- Language. Spec documents must be readable by an English-speaking reviewer: English-led or bilingual corpora (English structure and identifiers with other-language prose mixed in) qualify, and bilingual project pages say so. Projects whose spec prose is predominantly not in English are not tracked — readers could not verify our assessments against the primary sources.
- Exclusions. Two things disqualify a candidate regardless of how good its spec practice is. First, purpose: projects built to abuse or defraud another service — account farming, quota or trial circumvention, bulk redemption automation, spam — are not listed, because a directory entry reads as a recommendation. Second, star authenticity: watchers, forks, contributors, and issue traffic must scale plausibly with stars; when a star count looks promoted rather than organic, we treat the star floor as unmet.
- Editorial review. A maintainer reviews each surviving candidate and its evidence. The assessment may be researched and drafted manually or with AI assistance; nothing is published unreviewed.
Sweep results live in data/discovery/ in the public repository, machine-written like the
metrics. Some frameworks leave no public fingerprint at all — Augment Cosmos, for example, keeps specs and
agent state on its platform rather than in the repo — so their usage cannot be discovered this way.
Where the data comes from
Every page mixes two kinds of data, and the split is strict:
Automated metrics
Fetched by machines. Humans never edit them.
- What
- Stars, forks, contributors, open issues, repository age, push activity, releases, weekly commits.
- Source
- The GitHub API, via a scheduled job.
- Lives in
data/metrics/- Refreshed
- Daily. Each metrics panel shows its fetch date.
Curated assessments
Editorial judgments, published through review.
- What
- Framework descriptions, spec structure, drift ratings, defect and rework narratives, maintenance outcomes, timelines.
- Source
- Maintainers and contributors, sometimes using AI assistants, drawing on framework documentation, release notes, maintainer submissions, and verified case studies.
- Lives in
src/content/- Refreshed
- Best-effort review. Each page shows its "last reviewed" date; after 180 days the site marks the assessment as "Review due."
Both live in the same public repository, so every change — a bot commit or a human judgment — is reviewable in the git history. Stale assessments are a known failure mode: the "last reviewed" date tells you how fresh a judgment is. Assessments older than 180 days are marked "Review due" and can be found with the project directory's review-state filter. Flagging or refreshing one is one of the most useful contributions.
How to read the ratings
Three rating scales appear on cards and pages across the site. All three are curated editorial judgments, not computed values; the badges below are the exact ones you'll see.
Maturity — frameworks
How settled the framework's methodology is.
| experimental | Early stage; core workflow still changing; little real-world usage. |
|---|---|
| emerging | Real adoption and active development, but conventions still evolving. |
| established | Stable core workflow, meaningful adoption, actively maintained. |
| mature | Widely adopted, stable over multiple release cycles, proven in production use. |
Spec-to-code drift — projects
Whether the project's specs still describe its code, judged by comparing recent code changes against the spec artifacts.
| none | Specs are updated alongside code; spot checks find no divergence. |
|---|---|
| low | Minor gaps; specs trail code slightly but remain a reliable guide. |
| moderate | Specs noticeably lag; parts of the codebase are no longer described. |
| high | Specs are stale or abandoned while the code moves on. |
| unknown | Not yet assessed. |
Tracking status — projects
Whether the project is still being watched.
| active | Under regular development and regular review here. |
|---|---|
| paused | Development, or our tracking of it, is temporarily on hold. |
| archived | Finished or abandoned; kept for the historical record. |
Limitations
- Stars and commit counts measure attention and activity, not quality or productivity.
- Drift ratings are informed editorial judgments, which may be AI-assisted; they are not computed diffs and can lag reality between reviews.
- The sample is small and skewed: submitted projects come from teams proud of their process, and discovery sweeps only find frameworks whose tooling leaves public fingerprints in the repo.
- Frameworks differ in what they call a spec, so cross-framework comparisons should be read as directional, not precise.