The Confederation of African Football (Caf) said the team were flying home after Friday's gun attack on their team bus which killed three people.
As Angola prepared to play Mali in the opening game, reports claimed talks to persuade Togo to stay were continuing.
But as that match kicked off at 1900 GMT, Togo were heading for the airport in the northern province of Cabinda.
Togo's Prime Minister Gilbert Houngbo had repeated his insistence earlier on Sunday that the team should quit the competition over security fears.
Angolan President Jose Eduardo dos Santos condemned the attack in the tournament's opening ceremony in Luanda, and a minute's silence was observed.
The attack on Togo's convoy in the northern enclave of Cabinda killed an assistant coach, press officer and bus driver, and injured several other players.
Striker Emmanuel Adebayor, who plays in the Premier League with Manchester City, had previously said the team thought that life should go on and they did not want to be seen as ruining the tournament.
However, following further consultations with Togo government officials, he accepted the authorities' wishes.
He said: "This Friday at 1430, we were all dead on that bus. We sent our last messages to our families. We called our family to say our last words. I told myself: 'If you're still there on the ground in Angola, why not (play)?'
"The authorities decided we should return (home), so we will return."
Adebayor also revealed he spoke to members of the Ivory Coast and Ghana camps, two of the three other teams in their group - Burkina Faso being the fourth.
"They expressed their support by saying they were ready to leave the competition if we did," he said.
"(But) at the end of the day, we realised that they were ready to continue. It is still a continent where a World Cup will take place in South Africa.
"If we speak of the dead, the competition should have been cancelled. But the Confederation of African Football (Caf) has decided otherwise.
"We're going back and we wish good luck to those who will remain, especially to Burkina Faso, Ivory Coast and Ghana.
"What I have told their leaders is that they may be attacked at any time in Cabinda. I hope they will be cautious."
Team-mate Thomas Dossevi said part of the players' motivation for wanting to stay was to honour those killed, and French paper L'Equipe quoted another player Alaixys Romao as saying the team would not "leave like cowards".
Earlier, Houngbo said while he understood the feelings of the players and believed the competition should go ahead, the government stood by its decision to call the team back to Togo and had sent a plane to collect them.
"We took into account their last-minute change of mind," Houngbo told the BBC. "Up to midnight last night, they were all unanimous that they wanted to come back.
"After the delegation that we sent to visit them left, then they had another meeting and all of a sudden they changed their mind and we do respect that.
"But we explained to them through their captain, Adebayor, that in memory of those who had lost their lives, we also need to take into account what the families also think is the best way to pay tribute to them.
Houngbo criticised Caf, claiming it had not given Togo any information to assess the security situation after the attack.
"So far we did not even have a single call, even a call of sympathy, from Caf," he said.
"We do not even have information that will allow us to have an assessment from a security standpoint, taking into account what has happened."
Togo were due to play Ghana, who have confirmed they will remain in the tournament, in Cabinda in their first match on Monday.
The Togolese government has demanded Angola explain why it was not warned of the dangers of travel in Cabinda, where the bus was ambushed.
Angolan officials had earlier expressed surprise that the team had travelled there by road from their base in the Republic of Congo.
Togo's stance changed several times amid intense negotiations, with Dossevi quoted as saying that the players wanted "to show our national colours, our values and that we are men".
Caf had already said it understood Togo's initial decision to withdraw, but that the six other matches scheduled to be played in Cabinda would go ahead.
In Friday's attack, gunmen opened fire on the team bus shortly after it had crossed from the Republic of Congo into Cabinda.
Togolese officials said the driver had died at the scene, while media officer Stanislas Ocloo and assistant coach Amalete Abalo died later in hospital.
The separatist rebel Front for the Liberation of the Enclave of Cabinda (Flec), which has fought for independence for several decades but entered into a ceasefire in 2006, claimed responsibility for the attack.
CAF president Issa Hayatou said he had received a guarantee from the Angolan government that security would be increased for all teams and at all venues.
