Police questioned three of the gold medal winners and say "they were not victims of the crimes they claimed".
Two of the four - Gunnar Bentz and Jack Conger - were allowed to fly out of Rio on Thursday evening.
Ryan Lochte had already left, and James Feigen remains in Brazil.
Mr Bentz and Mr Conger had been taken off a plane at Rio airport on Wednesday and taken for questioning.
Mr Lochte had initially said the four were robbed at gunpoint when they were returning by taxi from a club.
But Rio's civil police head Fernando Veloso said the four Olympic gold medalists had not been robbed.
"No robbery was committed against these athletes. They were not victims of the crimes they claimed,'' he said.
He told reporters that one or more of the athletes had instead vandalised a toilet in a petrol station and then offered to pay for the damage.
The Americans paid and left after armed security guards intervened, he said.
One guard had justifiably drawn his gun after one of the swimmers began behaving erratically, Mr Veloso added.
He warned that the swimmers, who had repeatedly changed their accounts of what happened, could "in theory" face charges of giving false testimony and vandalism.
But later on Thursday, the men's lawyer Sergio Riera said Mr Bentz and Mr Conger had been allowed to leave the country by a special Olympic court.
"They are on their way to the airport," Mr Riera said.
However, Mr Feigen remains in Brazil.
He "provided a revised statement this evening (Thursday) with the hope of securing the release of his passport as soon as possible," said the US Olympic Committee (USOC).
The USOC statement confirmed the version of events given by Mr Veloso, and added that "the behaviour of these athletes is not acceptable, nor does it represent the values of Team USA".
Lochte is one of the most successful swimmers in history, with 12 Olympic medals, and he once had his own reality television show in the US. In Rio, he swam in two events, winning gold in the 4x200m freestyle relay along with Conger.
Feigen won gold in the 4x100m freestyle relay.
Bentz competed in the 4x200m preliminaries, but not the final. He still received a gold medal after the US team's win.