In a vote of Liberal Party members, Mr Sharma defeated Andrew Constance 251-206 in the final selection ballot on Sunday.
A former state treasurer and member for Bega, Mr Constance had been the firm favourite after being endorsed by the Opposition Leader Peter Dutton.
However, moderates within the party backed Mr Sharma who lost his lower house seat to independent Allegra Spender in the 2022 election.
Before entering parliament, he had served as Australia's ambassador to Israel from 2013 to 2017.
Mr Sharma said taking over from former senator Payne was a privilege.
"I would like to thank the party members for the opportunity to hold the Albanese government to account in the Senate over its many missteps and wrong decisions, and to fight for the many households across NSW struggling to deal with Labor's cost of living crisis," he said.
"The opportunity to serve in the Senate will allow me to fight for our nation's national security interests in a time of greater global turmoil."
Peter Dutton says Dave Sharma will "speak with moral courage and provide moral clarity". (Lukas Coch/AAP PHOTOS)
Mr Dutton said Mr Sharma's entry to the Senate would come at a crucial time.
"His diplomatic and foreign policy expertise will lend considerable weight and wisdom to the public policy debate given the precarious circumstances in Eastern Europe, the Middle East and the Indo-Pacific," the opposition leader said in a statement.
Deputy Opposition Leader Sussan Ley said Mr Sharma was a "fitting replacement" for Ms Payne.
"Dave Sharma will bring a unique perspective to the Australian Senate," Ms Ley said.
"Over the past 20 years, Dave has sat in the Oval Office with American presidents, helped to broker international peace agreements and has first-hand experience on-the-ground in Israel as a former ambassador."
Dave Sharma will be an outstanding Senator for New South Wales - an experienced, thoughtful and clever representative.— Senator Andrew Bragg (@ajamesbragg) November 26, 2023
Also passed over in Sunday's vote was former ACT senator Zed Seselja, a staunch conservative who hails from the right faction.
Mr Seselja lost his Senate spot to independent David Pocock at the last election, leaving the ACT without Liberal representation for the first time since the territory started with two senators in 1975.
Australia/Israel and Jewish Affairs executive director Colin Rubenstein said it was encouraging that in Mr Sharma the Senate would have another voice with a thorough and detailed knowledge of national security and world politics.
"He will hopefully help facilitate constructive, informed debate and dialogue and contribute to wise, effective decision making in Australia's national interest," Mr Rubenstein said in a statement.
Ms Ley said the Mr Sharma would be a great addition to the Liberal team as it worked to win back federal seats lost to independents.
"Dave is someone who very much appeals to that cohort of voters which we need to win in 2025," she said.
In another vote on Sunday, the Liberals pre-selected Kyle Hoppitt for the third spot on the party's Victorian Senate ticket.
