The key motivation of the study was to find out the process of social exclusion of resource-poor households (small, marginal, landless, and women farmers) in the study area of southern Bangladesh. The study adopted a qualitative approach, including case studies, focus group discussions, and key informant interviews to reveal the actual situation. A Fishbone diagram was adopted to represent the current social inclusion issues in the community. The study depicted that large affluent farmers were the protagonist in taking advantage of the capital-intensive agricultural intensification while resource-poor farmers were lagging behind. Polarization was visible between the rich and poor farmers at an increasing rate. Predominant factionalism and patron—client relations were observed in the power structure, but the encapsulation process was absent. Therefore, these issues constantly escalate socio-economic disparities pushing social exclusion with the initiation of agricultural intensification. Right-based awareness programs and livelihood improvement training should be promoted in the locality so that resource-poor farmers and the next generation can have benefited from these facilities.