The Wikipedia article on 빵 states that the word is borrowed directly from Japanese, and before that, from Portuguese:
From Japanese パン (pan, “bread”), from Portuguese pão (“bread”), from Latin panem, accusative singular form of panis (“bread”).
When did the word start being used in Korea, and what is the history of how it made its way into Korean? What was the Portuguese contact?
During the age of "discovery", the Portuguese were the first Europeans (see 1542) who "explored" Japan from as early as mid-16th century, spreading many European cultural and scientific components and their linguistic representations (see fourth paragraph)- of course in Portuguese - to Japan. I believe bread/Pão must be one of those.
Since then there were two major Japanese invasions of Korea - a medieval war at the end of 16th century that introduced many of those Portuguese "heritage" into Korea, for example, the Christianity and early muskets. Later, there was the imperialist colonization during the first half of the 20th century, terminated with the end of WWII. The Japanese ruling had a HUGE impact on the Korean language (that still exists) regarding almost everything related to the western culture and modern technology that was transplanted as a part of colonial industrialization or so-called modernization.
According to 'National Institute of Korean Language', it's from Portuguese.
Portuguese > Japanese > Korean. You can get more information here : NIKL Korean vocabulary history search 빵
From that website:
개화기 때 서양 문물이 유입되면서 ‘빵’도 같이 들어왔다. ‘빵’이란 단어는 한글 문헌에서 20세기에 와서야 나타난다. ‘빵’은 포루투갈어의 ‘pao’가 일본을 거처 한국에 들어오면서 ‘빵’이 되었다고 알려져 있다. 프랑스어로는 ‘pain’(빵)인데, 이것의 발음이 우리말 ‘빵’에 더 가깝다. 일본어에서는 ‘パン’(pan)로 차용하여 이것이 우리말로 건너 온 것이 ‘빵’이다. ‘빵’은 서양 사람들에게는 한국 사람의 ‘밥’과 같은 주식이다. 그래서 서양 격언에서 ‘사람은 빵만으로 살 수 없다’고 할 때의 ‘빵’은 사람의 먹을거리 전체를 의미한다.
