Hajime follows the bird rather than the people the bird is following, and so he too ends up a small distance away from that house. He's outside on the sidewalk, skulking and trying not to look too suspicious as he does so.