I had heard that in Korean, 애 used to be pronounced like 아이. This kind of makes sense as the glyph ㅐ looks like it's made out of ㅏ and ㅣ. In addition, in Japanese, some people pronounce the diphthong あい as え, such as in めんどくせー. I was wondering if this vowel change has a name or has been studied and how common it is. For example, in Mongolian, that diphthong is actually pronounced with kind of like an [æ] sound, so perhaps this vowel change isn't universal, given that there is a vowel change in the first place.
This is known as the monopthongisation of the diphthongs /aj/ and /əj/.
단(單)모음화 or 단모음화(單母音化).
In 1446, when hangeul was promulgated, what is now called Middle Korean had many more diphthongs:
This is readily visible from comparing 한자어 hanja-eo between historical Korean spelling, modern Sino-Korean pronunciation, and reconstructed Middle Chinese pronunciations, especially if you compare them with modern Standard Mandarin and Cantonese pronunciations. Most of the following is from Wiktionary:
海 바다 해 (sea)
Middle Chinese (fanqie): 呼改切
Middle Chinese (Zhengzhang Shangfang): /hʌiX/
Middle Korean hangeul: 해
Modern Korean pronunciation: 해ː /hɛː/
Standard Mandarin pronunciation: hǎi /xaɪ̯²¹⁴/
Standard Cantonese pronunciation: hoi2 /hɔːy̯³⁵/
世 인간 세 (world / generation)
Middle Chinese fanqie: 舒制切
Middle Chinese (Zhengzhang Shangfang): /ɕiᴇiH/
Middle Korean hangeul: 셰〯
Modern Korean pronunciation: 세 /se/
Standard Mandarin pronunciation: shì /ʂʐ̩⁵¹/
Standard Cantonese pronunciation: sai3 /sɐi̯³³/
對 대할 대 (to answer, to confront)
Middle Chinese fanqie: 都隊切
Middle Chinese (Zhengzhang Shangfang): /tuʌiH/
Middle Korean hangeul: (archaic hangeul alert) ᄃᆡ〯
Modern Korean pronunciation: 대 /tɛ/
Standard Mandarin pronunciation: duì /tweɪ̯⁵¹/
Standard Cantonese pronunciation: deoi3 /tɵy̯³³/
Note how the Middle Chinese ones are reconstructed with diphthongs or triphthongs, Middle Korean (which is significantly later than Middle Chinese) retains them, but Modern Korean has undergone monopthongisation. Of course, native Korean words also had these diphthongs (and even triphthongs).
When did these particular monophthongisations occur? Current consensus points to the late 18th century for the completion of the monophthongisation of these falling diphthongs in /j/, forming the new vowel phonemes /ɛ/ and /e/.
Through the 19th century, two other diphthongs also started to undergo monophthongisation, 외 and 위. But these new monophthongs, being front rounded vowels, were "typologically marked" (read: weird) and started to be diphthongised in the early 20th century, but with a different result than before - 외 /oj/ > /œ/ > /we/ and 위 /uj/ > /y/ > /wi/. Hence modern standard Korean has only one falling diphthong left, 의 /ɰi/, which has a very marginal status, and is likely a spelling pronunciation anyway.