If any of the files (or directories leading up to the files) have permissions set too loose, the connection will fail. Permission errors may be logged on the server side by the sshd(8) daemon.
Authentication refused: bad ownership or modes for directory …
In most cases, potential permission problems can be solved by restricting down access to the SSH configuration files. Permission changes to the home directory might be needed, though restricted rights may break other things.
server$ chmod go-w ~/
server$ chmod 700 ~/.ssh
server$ chmod 600 ~/.ssh/authorized_keys