Development programs have typically neglected uncertainty and variability in terms of outcomes and socio-ecological context when promoting conservation agriculture (CA) throughout sub-Saharan Africa. We developed a simple Monte Carlo-based decision model, calibrated to global data-sets and parameterized to local conditions, to predict the range of yield benefits farmers may obtain when adopting CA in two ongoing agricultural development projects in East Africa. Our general model predicts the yield effects of adopting CA-related practices average −0.60 ± 2.05 (sd) Mg maize ha−1 year−1, indicating a near equal chance of positive and negative impacts on yield. When using site-specific, socio-economic, and biophysical data, mean changes in yield were more negative (−1.29 and −1.34 Mg ha−1 year−1). Moreover, practically the entire distributions of potential yield impacts were negative suggesting CA is highly unlikely to generate yield benefits for farmers in the two locations. Despite comparable aggregate effects at both sites, factors such as land tenure, access to information, and livestock pressure contrast sharply highlighting the need to quantify the range of livelihood and landscape effects when evaluating the suitability of the technology. This analysis illustrates the potential of incorporating uncertainty in rapid assessments of agricultural development interventions. Whereas this study examines project-level decisions on one specific intervention, the approach is equally relevant to address decision-making for multiple interventions, at multiple scales, and for multiple criteria (e.g., across ecosystem services), and thus is an important tool that can support linking knowledge with action.