The banners had been hung from the team's balconies in the Olympic village, home to the 10,500 athletes from more than 200 countries who will compete at the Games starting next week.
"I have been informed that the banners have been taken down by the South Korean Olympic Committee," IOC President Thomas Bach told a news conference.
"This obviously happened on the request of the IOC because the guidelines are also saying very clearly that the Olympic village is one of the protected areas where the athletes can live peacefully together without having to face any kind of divisive messages," he said.
This is not the first time the two countries have been at odds over flags and symbols during the Olympics.
Despite having normalised ties in 1965, relations between the two countries have often been strained, largely over Japan's occupation of the Korean peninsula in the past.
At the 2018 winter Olympics in Pyeongchang, South Korea, the roles were reversed with Japan complaining over fans waving a Korean peninsula flag depicting disputed islands in the sea between the two nations known as Dokdo in Korean and Takeshima in Japanese.
South Korea also asked the IOC in 2019 to bar Japan from using the "Rising Sun" flag at the Games, a symbol of Japan’s imperialist past.