Earlier, Christian Cueva gave Peru a shock lead on Sunday, making the most of an error by Brazil goalkeeper Jefferson to open the scoring in the third minute.
However, it did not take Brazil – playing in their first competitive fixture since hosting the 2014 World Cup – too long to get back on level terms.
Peru had only themselves to blame as they inexplicably left Neymar unmarked in the box, allowing the Barcelona forward to turn home Dani Alves’ cross from the right wing after five minutes.
It seemed honours would finish even until Costa popped up in the box to steer home a winner as the game neared a conclusion.
The substitute’s dramatic strike extends Dunga’s unbeaten run in charge to 11 matches, and placing Brazil at the top spot in the group with Venezuela.
It looked like the Brazil of last year’s World Cup as calamitous defending led to the opener.
A harmless long ball led to David Luiz trying to dribble across his own six-yard box and it turned into a pass to goalkeeper Jefferson, who turned over possession trying a short pass of his own. Cueva was on hand to intercept before taking a touch and firing in from close range. But Brazil were level just two minutes later thanks to Neymar.
Alves, who caused plenty of problems down the right throughout the first half, crossed for his club colleague to head into the bottom corner. Neymar almost put Brazil ahead in the 13th minute, only to see his close-range effort cleared off the line by Carlos Zambrano.
The Brazil captain, who starred throughout, went close to making the breakthrough in the 53rd minute. Peru failed to clear despite two opportunities, and he whipped a 25-yard effort onto the crossbar.
Brazil looked the more likely to score and Costa squandered a great chance in the 75th minute, poking wide after Neymar put him through following a counter-attack.
Yet he made no mistake when given a second sight of goal, albeit when he had little of the net to aim at, after again being set up by the outstanding Neymar.