I'm afraid that my reaction to being grabbed like that by someone who wouldn't take 'no' would be to get out my mobile phone and start calling the police. Fortunately the British ones seem to accept 'no' quite easily, possibly because they know that they would be thrown off the premises by the owners if there were complaints.