This paper sets out to establish whether KAIZEN practices can contribute to the enhancement of service quality and productivity in the public sector in Tanzania. Currently, KAIZEN is still at its newest stage being applied in a few hospitals and very few Small Scale Manufacturing Enterprises (SSMEs).Using a narrative literature review approach, the study reviews 11 research-based articles to extract evidence on the applicability of KAIZEN in public service organisations. The study reveals that although KAIZEN originated in the manufacturing environment, its principles and practices translate well into other work situations including health services, public authorities, banks, education et cetera. The paper concludes that KAIZEN is applicable to the public service and it can drastically improve operational efficiency, service quality and reduce costs in that sector. The paper recommends that the government should adopt KAIZEN and incorporate it into all its public sector reform programmes as a strategic tool towards improvement of public service organisations’ overall effectiveness and performance.