Rafaelpta/dupehound

I reviewed the current main branch of dupehound by inspecting its Rust source, integration tests, documentation, benchmark harness, and CI/release workflows. This is clearly more than a README-only prototype. The architecture is coherent: gitignore-aware discovery, tree-sitter extraction for 14 languages, identifier and literal normalization, winnowing fingerprints, an inverted candidate index, exact Jaccard comparison, and union-find clustering. The scan, history, check, JSON, and MCP interfaces also make the core detector useful in both human and automated development workflows.

The repository has a meaningful test surface, including 75 Rust test functions, while its CI configuration covers Ubuntu, macOS, and Windows with tests, Clippy, and formatting checks. I also appreciate that the benchmark methodology documents important limitations instead of hiding them: the planted-clone ground truth, partial Type-4 support, the currently TypeScript-heavy real-repository experiment, limited agent runs, prompt sensitivity, and planned statistical and classical-detector comparisons.

My main recommendation concerns release security. The README promotes a curl-to-shell installer, but the release workflow publishes archives without checksums or signatures, and the installer does not verify the downloaded binary. Publishing SHA-256 checksums, verifying them before extraction, and eventually adding build attestations or an SBOM would make this installation path more trustworthy. Pinning third-party GitHub Actions to immutable commit SHAs would also strengthen the supply chain.

For evaluation quality, I would add broader per-language benchmark corpora and a classical duplicate detector baseline before implying that accuracy is equally established across all 14 supported languages. A published JSON Schema and compatibility policy would also help integrations depend safely on the versioned output.

Overall, dupehound is a focused and technically credible early-stage CLI with a real use case for AI-assisted codebases. Its deterministic, offline approach and CI-oriented “check before merging” workflow are especially strong. This review is based on source and workflow inspection; I did not execute the binary in my current environment.

Dupehound addresses a timely problem: agents can reproduce existing logic under new names because they do not inspect an entire repository before writing. It parses functions with tree-sitter, normalizes identifiers and literals, fingerprints structural token windows, and uses an index to find renamed copies deterministically and offline. The scan, history, and CI-focused check commands support exploration, trend analysis, and prevention, while explainable matches and MCP integration are practical additions.

The published planted-duplicate benchmark is useful and includes limitations, but broader evaluation on naturally occurring refactors would clarify precision and recall. Special cases such as generated code, framework boilerplate, required interface implementations, tests, and intentionally duplicated hot paths need transparent policies and regression fixtures. Similarity thresholds and grade labels should remain configurable so teams do not turn a heuristic into an inflexible quality gate. Dupehound is best for large or agent-heavy codebases that want fast, reproducible duplicate detection, especially when findings are reviewed as refactoring suggestions rather than treated as proof that code is bad.