Steyn said he chose to give up five-day cricket to prolong his career and he will continue to play limited-overs internationals.
"Today I walk away from a format of the game I love so much," Steyn said in a statement released through Cricket South Africa. "In my opinion test cricket is the best version of this game. It tests you mentally, physically, emotionally. It's terrible to consider never playing another test again but what's more terrifying is the thought of never playing again at all.
"So I will be focusing on ODIs and T20s for the rest of my career to maximize my full potential and ensure my longevity in this sport."
Steyn is contracted with the South Africa team for at least one more season.
The 36-year-old Steyn ends as South Africa's leading wicket-taker in tests and eighth on the all-time list with 439 wickets in 93 matches. Only four fast bowlers have taken more wickets than him in tests: James Anderson, Glenn McGrath, Courtney Walsh and Stuart Broad.
Steyn injured his right shoulder, his bowling arm, in a test in Australia in late 2016 and was out for more than a year. He then hurt his foot in his comeback test in 2018 and was sidelined for another six months and never quite recaptured his best form, although he did show glimpses of the bowler who was consistently ranked the best in the world in tests from 2008-14.
The shoulder injury raised fears he would never play again, but he returned to break the South African wicket record against Pakistan in December last year, overtaking Shaun Pollock as his country's most successful test bowler.
He made his debut in 2004 and his last test was at home against Sri Lanka in February.
Steyn hailed from the small mining town of Phalaborwa in remote northern South Africa and his raw talent ensured he became by far the town's most famous product.
He was a multi-skilled fast bowler who was fearsomely fast — and marked wickets with a down-on-one-knee fist-pumping celebration known as "the chainsaw" — but who could also beat batsmen with traditional out swing and reverse swing. His stats backed up his reputation and he has the best strike rate in test cricket among all bowlers who have taken more than 200 wickets.
"I'd like to thank everyone in cricket, no one specific, because everyone has been a part of my journey," he said.
Cricket South Africa chief executive Thabang Moroe said Steyn had "set the standard for our future generations to follow."