I know 도서관 is commonly used, but it's my first time hearing 서고 and the meaning is also library. Is there any differences between them?
On this point I think the Korea University Korean Language Dictionary makes good sense.
도서관: 온갖 출판물이나 기록물들을 모아서 보관해 두고 사람들이 이용할 수 있도록 한 시설.
서고: 책을 넣어 두는 방이나 집.
So, roughly translating, 도서관 is a facility where you store various publications and records for people to make use of it. 서고 is a room or a building where you store books.
To call a facility a "도서관", you would expect some level of publicity. Like a public library where anyone can come register and use, or a university library where any student of the school can come and use. Even if you say some 도서관 is "private", I would imagine a close-access facility where only those with special permission can enter for research, but not a very private room like you would expect no-one but yourself there. A bibliophile may have a 서고 in his own house, but hardly a 도서관. You can expect to find some video materials or newspapers at a 도서관, but the word 서고 does not imply that. If a 도서관 has more books than its open-access hall can house, it would put the other books in a closed-access 서고, or as the dictionary put it, "a room where they store books."
So I would say the more general translation of "a library" would be 도서관, but 서고 may fit only given the right context.
도서관 sounds like a typical library to me. You can literally mean a library with this.
서고 sounds like a repository where various kinds of documents or records are stored. However, I don't think 서고 is used often in casual conversation that much anymore. For such, I believe 기록보관실 or 자료보관실 is used more often these days.