And yeah there is some special sshd magic that will need to happen or I have to work to restrict ssh keys to specific locations in sshd