If you have made (and tested! very important to test!) backups for the wallet, then yes you can remove the wallet software from your device.
The thing is, if you ever want to use any wallet functions (eg, sending coins out, or generating new receiving addresses), then you’ll need some wallet software installed on some device to accomplish this.
To mitigate against ‘hacker’ type attacks, many users prefer using cold-storage (offline) hardware wallets. This style of wallet isolates your private keys in an offline environment, and it can then be paired with some wallet software that is running on an online device, in order to send/receive transactions (though this online wallet will not have access to your private keys, hence the safety enhancement).
Also worth noting that it is preferable to not only keep private keys offline, but to generate them offline in the first place. That way the user can have greater assurance that they will not fall victim to some malware/etc based attack due to a compromised device at the time of key generation.