ChatGPT can be tricked into writing a suicide note, designing a starvation diet for an anorexic or offering advice on how to hide intoxication at school.