This seems backwards, if he were still a duke he'd have land to worry about keeping in the family and would need to worry about his family's social standing, but they've already lost all that.
like sure still a valid concern if you have plans to claw your way back to power and he's rationalizing to disapprove of the relationship but this feels like we're supposed to take the logic at face value
The point is that if he had money and status, he could protect them by virtue of he said status, since he don't, he can't. It's still linked to him being overprotective of his daugther. He wouldn't mind too much otherwise.