Punishment infringes on one's rights - right of freedom (prison terms), right of property (fines and penalties), and various additional limitations (e.g.: right to vote, restrictions on where one may reside [sex offenders], etc). Immigration violations are essentially a contract matter: they came voluntarily and agreed to the conditions of their stay which they then violated.
Children being punished in a similar fashion: their freedom is taken away (grounded), their property is taken away (taking their phones/computer access/toys), their body is being violated (spanking etc, a crime on its own of course, but still happens).
Foreigner's presence in a country is not a right, it's a privilege. Host countries are sovereign and restricting that privilege is their sovereign right, not a punishment towards the foreigners. There are many foreigners being deported for many various violations of their conditions of stay even if these wouldn't be violations, and thus not punishable, for citizens.
Considering deportation to be punishment would bring into question considerations of jury trials, double jeopardy, burden of proof and the level of argument, etc.
While people may colloquially consider it to be punishment, the reality (not just "de jure", the actual reality) is that it is not. Foreigners are guests, even when they're permanent residents, and while they sometime forget that they are - they still are. Deportation is the restoration of order: before the offending guests came there was no offense. The guest offended and was asked to leave to restore a state without offense. The guest's rights are not infringed. Their liberty is not restricted (other than ensuring their departure), and their property is still theirs. They can say whatever they want, so their freedom of speech is similarly retained. Just say it elsewhere.