Most likely there is a monetary issue here. The owner will be better off if this house is not made into a landmark. Accusing someone of being a Nazi is an easy way to discredit someone.
Of course the average knuckle-dragging human falls for this type of slander the majority of the time.