Android (e.g. Samsung, HTC, Sony): To cancel an invitation, open the group settings (open the menu by tapping on the group name) and delete the invitation by tapping the x after the person's name.
iOS (iPhone): To cancel an invitation, open the group settings (open the menu by tapping on group settings) and tap the minus in the red circle after the person's name, then tap delete.