A dog most likely interprets a smiling person as baring their teeth, which is an act of aggression.

As an extra, here's a cool little doggo.