On Assessing the Value of a Project

In my experience, a small fraction of research projects produces most of a team’s impact. I learned the following framework from Mu Li several years ago, and my team uses it to compare projects before committing people or compute.

The expected impact of a project can be approximated by three factors:

1. the probability that it meets a stated success criterion, $\Pr(P)$,
2. its positive fractional effect on the chosen outcome if it succeeds, $ES(P)$,
3. its weighted applicable scope, $AS(P)$, which captures how often or how broadly the result can be used.

Putting it together:

$$\text{Expected impact}(P) \;\propto\; \Pr(P) \cdot ES(P) \cdot AS(P)$$

This product is a planning approximation, not a literal probability model. If effects and success rates vary substantially by scenario, estimate them separately:

$$\text{Expected impact}(P) = \sum_s \Pr(\text{success}\mid s)\,ES(P,s)\,W_s,$$

where $W_s$ is the exposure or importance of scenario $s$, and the weights use one consistent scale across the projects being compared. Counting scenarios without weights is easy to game by splitting one use case into many. The product form is useful when a single probability, effect, and reach estimate summarize the project well enough for comparison.

The framework estimates upside, not cost. I compare the product directly only when projects require similar resources. Otherwise, I divide by the constrained resource—usually researcher time, accelerator time, or calendar time—and also consider the opportunity cost of delaying other work.

Expected impact ∝ 25.0

Pr(P) — chance of success0.50

01

ES(P) — positive fractional effect5.0

010

AS(P) — weighted reach10

020

Estimating chance of success — Pr(P)

Before estimating probability, define the minimum result that counts as success: a target metric, operating condition, and deadline. Projects fail through risks we did not identify and risks whose outcomes we could not predict. A broader scope can contain more of both. Three practices improve the estimate:

Review prior work. A literature survey maps known failure modes, how others approached them, and the outcomes they observed. Record the closest prior attempts and the assumptions that differ from the proposed project; these differences identify which failures remain informative and which may not carry over.

Reduce complexity. State the project’s assumptions, test them early, and remove requirements that do not affect the core idea. Each removed dependency can eliminate one failure mode, provided that removing it does not introduce a new assumption elsewhere.

Record predictions. Preserve the estimate made at decision time so the team cannot rewrite its original rationale after seeing the outcome:

1. Record a numerical probability, the key assumptions, and the supporting evidence.
2. Do not revise the recorded estimate after observing the outcome.
3. Track mean Brier score across completed binary predictions. When checking calibration by probability bucket, use enough observations in each bucket to make the comparison meaningful.

Deriving effect size — ES(P)

Define effect so that larger is always better. For a lower-is-better metric $x$, use $(x_{\text{baseline}}-x_{\text{project}})/x_{\text{baseline}}$; reverse the numerator for a higher-is-better metric. A 30% reduction is therefore an effect size of 0.3.

Incomplete information makes absolute effect difficult to estimate, but an upper bound is often enough to reject a project. Assume the project eliminates its target bottleneck, then identify the factors it cannot change. For example, if a kernel accounts for 30% of training time, eliminating its runtime entirely can reduce end-to-end time by at most 30%. Communication, input stalls, and the remaining kernels set that bound.

Use the same outcome metric across projects being compared: end-to-end training time, loss at a fixed budget, accelerator-hours, revenue, or another measurable quantity. Dollars are useful only when the conversion is credible.

Understanding applicable scenarios — AS(P)

Typically, a project applies where its assumptions hold. Removing an assumption can increase reach, but supporting the more general case may add implementation or research risk. It may also reduce the effect if the broader solution cannot exploit properties of a narrower workload. These are common tradeoffs, not fixed laws, so re-estimate all three factors whenever the scope changes.

For a new direction, I usually begin with a narrow scenario whose assumptions can be tested quickly. If it succeeds, I relax one assumption at a time and recompute probability, effect, reach, and cost before expanding the project. The narrow result has immediate value and records which assumptions survived contact with evidence, making the next generalization decision better informed.