Consider having an option to let the user specify public key(s) to register for ssh rather than taking all of them in ssh-agent.