Mr Amano won the required two-thirds majority from the IAEA governing board, defeating his South African rival, Abdul Samad Minty, by 23 votes to 11.
The present head, Mohamed ElBaradei, is scheduled to end his term in November.
A previous round of voting by the UN nuclear agency's 35-nation board in March was indecisive.
Mr Amano had the support of many industrialised countries while Mr Minty was favoured by most developing nations.
The vote reflected that split, with Mr Amano scraping through to victory by the bare minimum margin, without the support of developing states. There was also one abstention.
Mr Amano has held senior arms control posts in Tokyo and most recently was Japan's representative at the IAEA.
Correspondents say his narrow victory may weaken his position, as many countries had stressed the need for the new head to be chosen with the broadest possible backing, to be able to tackle the threat of nuclear proliferation.
Some diplomats see Mr Amano as less political than the outgoing chief, Mr ElBaradei who, they say, has been too soft on Iran and too ready to speak out on matters outside his mandate.
The three terms of Mr ElBaradei, who has been in office since 1997, were marked by rows with the George W Bush administration over Iraq and Iran.YUKIYA AMANO