Introduction: To estimate differential accumulation of soil organic carbon (SOC) and its ecological significance is very important to smallholder farmers in the Tanzania’s semiarid areas. This study investigated the accumulation of SOC and other important soil nutrients under organic and no-fertilizations, and correlated SOC with crop yields. Using data from long-term experimental study sites of semiarid in Tanzania, we estimated SOC accumulation in different soil treatments and depths where a total of 128 soil samples were collected at the depths of 0–20 cm and 20–40 cm from two villages (sites) with organic fertilization and nofertilization treatments. Sites under organic fertilization were defined as those which have received manure fertilization for more than 5 years on continuum basis. Outcomes: The accumulation of SOC was significantly greater in soils under organic fertilizations (1.15 and 0.80 MgC ha−1 at soil depth and 20–40 cm) and decreased with increasing soil depths. Similarly, TN and P decreased from 0.40 and 2.40 Mg (0–20 cm) to 0.16 and 2.10 Mg (20–40 cm), respectively. Other important soil nutrients such as calcium (Ca2+), potassium (K+), magnesium (Mg2+), and sodium (Na+) had similar pattern. In addition, soil bulk density was less under organic fertilization (1.1 g/cm3) than under no-fertilization (1.2 g/cm3) and it increased with soil depths. Correspondingly, the crop yields were significantly higher (1.6 tn ha−1) under organic fertilizations than (0.6 tn ha−1) under no-fertilizations indicating that crop yields were significantly affected by SOC. Discussion: Organic fertilization i.e., especially cattle manure in the area has considerable fertility potential. To optimize soil fertility potential, we need to consider such kind of fertilization from household to national level. Conclusion: Our results demonstrated that manure application was the best fertilization method for improving soil fertility in most croplands of Tanzania’s semiarid areas, especially in this era of climate change scenarios.