"Maybe, but they'll also remember your face because there was nothing else to look at."

There are really unassuming people that are easily forgotten, but Hajime's face is ...it has too many specifics. Unless, perhaps, someone thinks that all people of ethnicity X look the same. But trusting in that is stupid.