A couple things you can check:
If you are using the AllowUsers directive make sure the IP/hostname matches what's defined in sshd_config.
If you are using hostnames instead of IP addresses with the AllowUsers directive verify the hostname reverses properly.