If he's bothered at all, he doesn't show it. Probably thinking kids to himself in a very put-upon voice. "People would still hear an invisible dragon, don't you think? And a trapdoor, they'd still have to come out sooner or later."

Hence why he's watching, or so indicates his gesture towards the construction site.