Maybe he knows a thing or two about kami that he can't tell anyone, because if he did, they'd think he's insane? He certainly seems a bit too shaken for it to only be about image. Then again, Akiko herself worried a lot about her image earlier, so perhaps they share a deep fear of not living up to certain expectations?