A second half try by cardiff's Harry Robinson, converted by Dan Biggar,was the crucial score for Wales. Biggar kicked four penalties and replacement Rhys Patchell also kicked a penalty on his debut.
Japan scored tries through Michael Broadhurst and Yoshikazu Fujitsu, with full-back Ayumu Goromaru converting one and kicking two penalties.
Goromaru actually gave Japan a 6-0 first half lead with two penalties as Wales struggled up front. Number eight Rob McCusker was yellow carded but Biggar responded with two penalties of his own.
However, Japan went into half time with a 11-6 lead as flanker Broadhurst powered over untouched from a well worked lineout move.
After the break, Goromaru was crucially off target with two penalties before Biggar showed him the way and nudged over two penalties to edge Wales into a 12-11 lead on the hour.
Then came Robinson's key try, as full back Liam William's pass gave him the room to cross.
Japan made things stressful for Wales as wing Yoshikazu Fujita crossed on the right, with Goromaru converting.
But Rhys Patchell's late penalty calmed the nerves and clinched the 22-18 win.