The other two answers provide partial insight, but not the whole story. Let me combine those two and provide some more background.
언니 was indeed used to refer to a male's older brother as well, as @jick says, but this word only starts to appear in records in the late 19th
century. There is no evidence that this word was used before that time.
Then, what was used instead? The answer is: "형". "형" is now only used to refer to a male's older brother, but in the past, it was used to refer to a female's older sister as well.
Basically, "형" used to mean "one's older sibling who has the same gender". This usage is fossilized in the term "형님", which is also used by females to address their older sister-in-law.
Here's an instance of "형" being used in Middle Korean that is used to refer to a female's older sister, in <석보상절 권6> from 1447:
아ᄌᆞ〮마니ᄆᆞᆫ〮 大땡〮愛ᅙᆡᆼ〮道또ᇢ〯ᄅᆞᆯ 니르시〮니〮 大땡〮愛ᅙᆡᆼ〮道또ᇢ〯ㅣ 摩망耶양夫붕人ᅀᅵᆫㅅ 兄ᄒᆑᇰ니〯미〮시니〮
야ᇰᄌᆡ〮 摩망耶양夫붕人ᅀᅵᆫ만〮 몯〯ᄒᆞ〮실ᄊᆡ〮 버근〯 夫붕人ᅀᅵᆫ이〮 ᄃᆞ외시〮니라〮
Translated word-for-word to Modern Korean, the above would be:
"아주머님"은 대애도(大愛道)를 이르는데, 대애도가 마야부인의 형님이시니, 인물이 마야부인만 못하신 까닭에 둘째 부인이 되셨다. 〈석보상절06:1a-1b〉
In English:
The "aunt" refers to Pajapati. Pajapati is Queen Maya's older sister. Her appearance was not as beautiful as Queen Maya, so she become the second wife.
Then, where does the word "언니" come from, if it just suddenly appeared in the 19th century? There are a lot of theories, but the most convincing one is that it is a corruption of the word "형님" from "형(兄)" (same-sex older sibling) + "-님" (honorific). There are several evidences that support this theory, but I won't go into that here.
In Middle Korean, the kinship terms of siblings were quite different from how they are now. As I said before, 형(兄) was used for "same-sex older sibling", but what about others? I summarize those in the table below:
|
|
male's |
female's |
older |
brother |
형 |
오라비 |
|
sister |
누의 |
형 |
younger |
brother |
아ᅀᆞ |
오라비 |
|
sister |
누의 |
아ᅀᆞ |
In short, there were four "basic" sibling kinship terms in Middle Korean:
- "형": "same-sex older sibling"
- "아ᅀᆞ": "same-sex younger sibling"
- "오라비": "female's (older/younger) brother"
- "누의": "male's (older/younger) sister"
Out of those four, only one, "형(兄)" is Sino-Korean. Why is this so? I don't know. What was used to refer to "same-sex older sibling" before 兄 was loaned into Korean? I don't know, but some people suggest that "맏이" (which now means "firstborn" or "eldest sibling") originally referred to "same-sex older sibling", although there is no evidence.
When did 동생(同生) "younger sibling" begin to be used? 동생 (도ᇰᄉᆡᇰ in Middle Korean) did exist in Middle Korean, but it originally used to mean "siblings coming from the same mother" (as opposed to half-siblings: siblings who have different mothers). This is evident from the Hanja: 동(同) "same" + 생(生) "born".
Through a semantic shift process, 동생 now means "younger sibling", and these supplanted the earlier "아ᅀᆞ", "누의", and "오라비". (But "누의" and "오라비" kept living on restricted to "older" siblings, becoming "누의 + 님 > 누님 + 아 > 누나" (male's older sister) and "오라비 + -아 > 오빠" (female's older brother))