Adding to the two great answers posted by @ClaraKim and @jick, you need to note that the exact opposite word of '바깥사람 (literally, outside person)' could be '안사람 (inside person)' rather than '집사람 (house person)'. It will depend on how people react to the words, but '안사람' sounds more respectful than '집사람' as the latter sounds like a worker at home.
One strange thing is '바깥사람' could be changed to '바깥분' as an honorific, but '안사람' is not changed to '안분' and sometimes '바깥양반 (양반 means a noble person in Chosun Dynasty)' is used as a more honorific word than '바깥분', but there is no such word as '안양반'. Sexism in Korean? Yes.
'안사람', '집사람' all are used to replace '아내' or '부인'. '바깥사람', '바깥분', and '바깥양반' all mean '남편' and who takes care of household chores and who earns money outside home will not change the usage unless you want to make a joke using the words.