Peter would recognize his own father for sure. Also the age of the invisible man is just slighty older then Peter so the Mother cheating on her husband theory wouldn't play into it either. I do however think that the Mother has powers that haven't been shown yet.
