N-gram Language Models
0 likes68 views
The document discusses riverbank erosion in Bangladesh, highlighting its severe impact on displacement, destruction of livelihoods, and socio-economic impoverishment for the affected populations. It emphasizes the lack of effective policies for rehabilitation and the challenges faced by 'environmental refugees' compared to those displaced by political violence, who receive more assistance. The findings draw from a social survey conducted in 2008, detailing the plight of displaced individuals and the necessity for targeted interventions to address the ongoing crisis.
N-gram Language Models
- 1. Khulna University of Engineering & Technology Department of Computer Science and Engineering Course Name: Natural Language Processing Course No.: CSE 4221 Assignment on: N-gram Language Models Submitted to: Submitted by: Dr. K. M. Azharul Hasan Suman Mia Professor Roll: 1607045 Dept. of CSE, Year: 4 th KUET Term: 2 nd
- 2. 1. The genre is “About Riverbank Erosion” 2. Collecting a corpus with more than 100 sentences: ( Ref: https://www.researchgate.net/publication/265077000) Environmental refugees are one of the most burning issues at this time throughout the world. Bangladesh, a riverine country, is suffering from acquit riverbank erosion which compels millions of her population to be displaced from their place of origin. As such, 283 locations, 85 towns and growth centers, along with 2400 kilometers of riverbank line in Bangladesh are vulnerable to erosion. Riverbank Erosion an endemic and recurrent natural hazard in Bangladesh. The major rivers e.g., the Padma, the Jamuna, and the Meghna, erode several thousand hectares of floodplain making thousands of people landless and homeless every year. Along with the floodplain, Bangladesh loses several kilometers of roads, railways, and flood-control embankments annually. No other disaster is as disastrous as riverbank erosion and ‘Internally Displaced Populations’ (IDP) face many unavoidable problems at different stages of displacement. Displacement marginalized them in respect of livelihood patterns and psycho-physical troubles. Such forty million homeless people in Bangladesh are compelled to lead a floating life. Riverbank erosion plays a major role in socio-environmental changes. The displaced people of riverbank erosion experience substantial socio-economic impoverishment and marginalization as a consequence of involuntary displacement from their original residence. Findings of a social survey carried out in 2008 on the erosion-hit displaced people in Chapai Nawabganj and Rajshahi districts are discussed in this paper. Newly settled people along with the native inhabitants have been interviewed to reveal the problems associated with making rearrangements for the displaced people. Besides these problems of displacement, adaptation strategies and relationships with the local people, their socioeconomic losses, sufferings, perception of natural disaster and psycho-physical problems, environmental ruins etc. have been revealed in this paper. There are no specific policies to rehabilitate the erosion-hit people. Thus, it is time to formulate policies to address prevention of riverbank erosion as well as to rehabilitate the river-erosion refugees. Bangladesh is a land of rivers. More than 700 rivers, with their tributaries and distributaries have criss-crossed the country forming a network of river system. It has about 2,400 kilometers of bank line and along with the bankline there are 283 locations, 85 towns and growth centers are vulnerable to erosion1. The Padma (The Ganges), the Jamuna and the Meghna, major rivers of Bangladesh, erode several thousand hectares of floodplain, making thousands of people landless and homeless every year. In recent years, throughout the world, victims of environmental disasters are growing in number than the victims of war or other (man-made) forceful migration. Victims of such political violence receive satisfactory media coverage along with different types of assistance from various national and international agencies. But the uprooted migrants- caused by environmental disasters- fail to attract such level of local or international attention which tends them to be more vulnerable. According to the United Nations University (UNU), victims of political violence receive a lot of assistance from governments and international organizations in forms of financial grants, food, tools, shelter, schools and clinics, but
- 3. “environmental refugees” have not such access to receive aid.2 An empirical study conducted among the riverbank erosion displaces of Rajshahi and Chapai Nawabganj, two districts of Bangladesh, by writers in 2009. The respondents were displaced due to erosion of the Padma and the Mahananda, two rivers of the said two districts. Report of the Climate Institute published in 1995, says that a growing number of people are displaced as a result of environmental problems such as drought, soil erosion, desertification and deforestation.6 It has further estimated that about 3.3 million people in Indonesia and 7.1 million in India are at risk from flooding and inundation by 2070. It also expressed awareness of large-scale displacement in Bangladesh caused by sea level rising and coastal or riverbank erosion. Riverbank erosion displacees face few unavoidable problems in different times of displacement, i.e. before displacement, during shifting household materials and family members and after displacement at their new settlement area. Displacee households were living in a settled area from generation to generation. Due to riverbank erosion, they were forced to migrate from their place of origin to other places, mostly unknown to them. Displacement due to riverbank erosion marginalized them in respect of livelihood patterns and psycho-physical troubles. In Bangladesh Riverbank erosion people lose their land and become homeless. Only the difference is, death counts less in number. Riverbank erosion displacees’ losses are unbound. There are few losses, which can never be converted to money. Besides the loss of land, they lose other things too. Being homeless, they become asset less too. People of the study area have lost their agricultural and homestead land in one hand and on the other hand they became rootless, ousted from their community, broken down their family ties and social bondage. The effect was enormous and the loss was quite impossible to regain. The primary cause of river bank erosion is flowing water. Riverbank erosion aggravates land of the riparian inhabitants. It was found that, all of the respondents of study area had lost their land, either totally or partially. They were displaced from their original homestead, which was an unwilling displacement. Impact of unwilling displacement is always negative, either economically or socially. Land is fundamental for civilization, to build home and establish family followed by a society. From the study it was found that loss of land, as well as displacement, was the only reason for their present sufferings and other problems. They expressed that they followed their own strategy and system to mitigate needs and problems. But due to displacement and settling in a new area, things were out of their reach. Searching for homestead land turned into priority and a few of them managed to become landowner. A number of respondents were found, determined not to make settlement in their present living place. Due to the geographical location, the country frequently suffers from devastating natural hazards of which, floods, cyclones, tornadoes, riverbank erosion, drought and earthquakes are the most disastrous to mention. Flood, cyclone, tornado and drought are seasonal disasters in Bangladesh. Earthquakes happen rarely in this country. But only the riverbank erosion is recurrent and it happens throughout the year. Riverbank erosion is a unique type of environmental disaster due to its destructive nature. In other disasters, resources may loss or people may displace for a short period of time but land
- 4. exists. On the other hand, land, the primary component of civilization, is washed out, displacement occurs forever. So, no other disaster is as disastrous as riverbank erosion. Unfortunately, it was not considered a specific disaster in Bangladesh before1993, rather thought to be a consequence of flood. But flood is not turns always a disaster. After continuous campaign and demand from concerned agencies and stockholders, in 1993, government of Bangladesh declared riverbank erosion as a disaster. Though riverbank erosion is a unique type of disaster but it is amalgamated with other disasters and affected people are entertained on the basis of the existing policies and programs of disaster management. The National Policy on Disaster Management has emphasized a group of broad based strategies, i.e. risk management, community involvement and non-structural mitigation measures. In Bangladesh there are three bodies for multi-sectoral coordination and collaboration at the national level26. a) The National Disaster Management Council (NDMC) is headed by the Prime Minister, b) Inter Ministerial Disaster Management Coordination Committee (IMDMCC), headed by the cabinet Minister in charge of the Ministry of Food and Disaster Management (MFDM) and c) National Disaster Management Advisory Committee (NDMAC) with memberships from both the public and private sectors. There are committees also in district, upazila and union level followed by the directives of national committees. Riverbank erosion displacees do not get first two types of assistances because of the required qualifications. For getting CI Sheets, victims have to have their own land. But ‘displacees became displacees’ after losing their land due to riverbank erosion. So, they fail to get CI Sheets that is essential for building a temporary shelter. They do not get general relief (cash) also. If a person dies due to disaster then his family may get assistance as cash. Fortunately, death is rare in case of riverbank erosion victims. Twenty KGs of rice is distributed to each family only for once as general relief, with which a five-member family can go maximum ten days. Three thousand Taka is given to each family to repair their damage houses. Problem also lies in selecting victim families. There are temporary shelters build by the government at cyclone prone areas, where people take shelter during cyclone-surge. Authorities can easily get reach to them. They also can reach to those who do not come to the shelter, because they live in their own land. It is also the same for the flood affected people. But there is no temporary shelter for riverbank erosion victims or no early warning system or no early evacuation process to evacuate the inhabitants of erosion prone areas. As a result, after losing homestead, displacees leave the area on their own initiatives to distant safer places as there is no chance for early alluvial of their diluviated land. Field level experience shows that most of them move to different administrative area. So, the officials of their original area cannot assist them and the officials of resettled area have limited scope to help them as they are not the victims of that administrative area. The consequence is that, these environmental refugees become more vulnerable in every sense and compel to live under the extreme poverty line.
- 5. 3. The input sentence In Bangladesh Riverbank Erosion is one of the ……… 4. The output sentence: In Bangladesh Riverbank Erosion is one of the respondents. 5. For applying bigram model; We have to search the corpus and find the word used after ‘the’ with highest probability. The words after the word ‘greatest’ are: respondents (3), Meghna (3), erosion (2), river (2) and 25 different word So, ‘respondents’ has higher probability than others in the input sentence using MLE. By Maximum likelihood estimation, Probability (respondents | the) = (3/43) = 0.06976 For bigram model, the input sentence is In Bangladesh Riverbank Erosion is one of the respondents.
- 6. Table 1. Raw bigram count table for the input sentence In Bangladesh Riverbank Erosion is one of the respondents In 0 6 0 0 0 1 2 0 0 Bangladesh 0 0 0 0 1 0 0 0 0 Riverbank 0 0 0 15 0 0 0 0 0 Erosion 0 0 0 0 2 0 1 0 0 is 0 0 0 0 0 1 0 0 0 one 0 0 0 0 1 0 1 1 0 of 0 0 0 0 0 0 0 7 0 the 0 0 0 0 0 0 0 0 3 respondents 0 0 0 0 0 0 1 0 0 Table 2: normalized by unigram In Bangladesh Riverbank Erosion is one of the respondents 26 11 16 23 9 3 50 43 3 Result: In the corpus, we have 933 unique words, so, V= 933 Using add- 1 estimation for smoothing, P (Bangladesh |In)= (6+1)/ (26+933) = 0.00723 P (is | Erosion) = (2+1)/ (23+933) = 0.00313 and so on Table 3: Raw bigram probabilities In Bangladesh Riverbank Erosion is one of the respondents In 0.001042 0.0073 0.00104 0.00104 0.00104 0.00208 0.00312 0.00104 0.00104 Bangladesh 0.00105 0.00105 0.00105 0.00105 0.0021 0.00105 0.00105 0.00105 0.00105 Riverbank 0.00105 0.00105 0.00105 0.0168 0.00105 0.00105 0.00105 0.00105 0.00105 Erosion 0.00104 0.00104 0.00104 0.00104 0.00312 0.00104 0.00208 0.00104 0.00104 is 0.00106 0.00106 0.00106 0.00106 0.00106 0.00121 0.00106 0.00106 0.00106 one 0.00106 0.00106 0.00106 0.00106 0.00213 0.00106 0.00213 0.00213 0.00106 of 0.00107 0.00107 0.00107 0.00107 0.00107 0.00107 0.00107 0.00712 0.00107 the 0.00102 0.00102 0.00102 0.00102 0.00102 0.00102 0.00102 0.00102 0.0041 respondents 0.00106 0.00106 0.00106 0.00106 0.00106 0.00106 0.00213 0.00106 0.00106
- 7. So, P (In Bangladesh Riverbank Erosion is one the respondents) = P (Bangladesh |In) *P (Riverbank |Bangladesh) *P (erosion | Riverbank) * P (is |erosion) *P (one | is) * P (of| one) *P (the | of) *P(respondents| the) = 0.0073*0.00105*0.0168*0.00312*0.00121*0.00213*0.00712*0.0041 = 3.02*10^-20(Ans) 6. For applying trigram model, We have to search the corpus and find the word used after ‘the greatest’ with highest probability. The words after the words ‘of the’ are: most (1), respondents (2), climate (2), said (1),risparian(1). So, by MLE, P (respondents | of the) = (2/7) = 0.285 For trigram model, the input sentence is In Bangladesh Riverbank Erosion is one the respondents. Table 4. Raw trigram count table for the input sentence In Bangladesh Riverbank Erosion is one of the respondents In Bangladesh 0 0 2 0 0 0 0 0 0 Bangladesh Riverbank 0 0 0 2 0 0 0 0 0 Riverbank erosion 0 0 0 0 1 0 0 0 0 Erosion is 0 0 0 0 0 1 0 0 0 Is one 0 0 0 0 0 0 1 0 0 One of 0 0 0 0 0 0 0 1 0 Of the 0 0 0 0 0 0 0 0 2 The respondents 0 0 0 0 0 0 1 0 0 Table5: Normalized by unigram In Bangladesh Bangladesh Riverbank Riverbank erosion Erosion is Is one One of Of the The respondents 6 2 3 2 1 1 7 4 Result: In the corpus, we have 933 unique words, so, V= 933 Using add- 1 estimation for smoothing, P (Riverbank| In Bangladesh) = (2+1)/ (6+933) = 0.00327 and so on
- 8. Table 6: Raw trigram probabilities In Bangladesh Riverbank Erosion is one of the respondents In Bangladesh 0.00106 0.00106 0.00327 0.00106 0.00106 0.00106 0.00106 0.00106 0.00106 Bangladesh Riverbank 0.001067 0.001067 0.001067 0.0032 0.001067 0.001067 0.001067 0.001067 0.001067 Riverbank erosion 0.001068 0.001068 0.001068 0.001068 0.00213 0.001068 0.001068 0.001068 0.001068 Erosion is 0.001069 0.001069 0.001069 0.001069 0.001069 0.00214 0.001069 0.001069 0.001069 Is one 0.00107 0.00107 0.00107 0.00107 0.00107 0.00107 0.00214 0.00107 0.00107 One of 0.00107 0.00107 0.00107 0.00107 0.00107 0.00107 0.00107 0.00214 0.00107 Of the 0.001063 0.001063 0.001063 0.001063 0.001063 0.001063 0.001063 0.001063 0.00319 The respondents 0.001067 0.001067 0.001067 0.001067 0.001067 0.001067 0.00213 0.00106 0.001067 So, P (In Bangladesh Riverbank Erosion is one the respondents) = P (Riverbank |In Bangladesh) *P (erosion | Bangladesh Riverbank) * P (is | Riverbank erosion) *P (one |erosion is) * P (the | is one) *P (the |one of) *P (respondents| of the) = 0.00327*0.0032*0.00213*0.00312*0.00214*0.00214*0.00214*0.00319 = 6.96*10^-19(Ans) 7. For applying unigram, The output sentence: In Bangladesh Riverbank Erosion is one the respondents. Table 7: Raw unigram count table In Bangladesh Riverbank Erosion is one of the respondents 26 11 16 23 9 3 50 43 3 As there is no zero value is the count table, no smoothing algorithm is needed Result:
- 9. The total valid words in the corpus are: N = 1987 So, P(In) = (26/ 1987) = 0.013 P(is) = (9/1987) = 0.00452 Table 8: Raw unigram probabilities In Bangladesh Riverbank Erosion is one of the respondents 0.013 0.00553 0.00805 0.0115 0.00452 0.0015 0.025 0.021 0.00151 So, P (In Bangladesh Riverbank Erosion is one the respondents) = P (In) *P (Bangladesh) *P (Riverbank) * P (erosion) *P (is) * P (one) *P (of) *P(the) *P (respondents) =0.013*0.00553*0.00805*0.0115*0.00452*0.00151*0.025*0.021*0.00151 =3.35*10^-20(Ans) 8. Discussion of the result: Final output sentence is: In Bangladesh Riverbank erosion is one of the respondents. Table9: Comparing probabilities for the final input sentence Unigram Bigram Trigram Probability 3.35*10^-20 3.02*10^-20 6.96*10^-19 As seen in table 9, for the final input sentence (In Bangladesh Riverbank erosion is one the respondents.) I got the highest probability (6.96*10^-16) in trigram model. So, trigram model is giving better prediction than unigram model for the final input model.