The 67-year-old was shot and killed while speaking near a train station in the Japanese city of Nara on Friday. A man was arrested at the scene.
AOC President Ian Chesterman has praised Mr Abe as a dear friend of the Olympic movement and says the success of the Tokyo Games during the COVID pandemic was a tribute to his leadership and perseverance.
"We are deeply saddened by the loss of Mr Abe," Mr Chersterman said on Saturday.
"The Tokyo 2020 Olympic Games were such a triumph, giving hope and inspiration to so many around the world."
IOC Vice President John Coates worked closely with the then prime minister in his capacity as chair of the IOC Coordination Commission for the Tokyo Games.
"Without Shinzo Abe's commitment, the decision to postpone the Games by one year in the face of the global pandemic would simply not have happened.
"All of the athletes of the world who participated in the Tokyo 2020 Games are forever indebted to him," Mr Coates said.
The Australian business community has also described Mr Abe as a visionary leader in the Indo-Pacific.
Business Council chief executive Jennifer Westacott says his transformative economic program helped pave the way for a reinvigorated Japanese economy and greater prosperity for his people.
"As a champion of a freer and more open world, the legacy of Mr Abe's leadership is greater co-operation of all like-minded nations and a more secure region," she said.
"At a time when the world needs strong, visionary leaders more than ever ... Mr Abe's loss will be sorely felt across the globe."
Prime Minister Anthony Albanese on Friday said Mr Abe was one of Australia's closest friends and that under his leadership Japan had emerged as one of Australia's most like-minded partners in Asia.