Source: IT Law Firm

Generative AI systems can produce outputs that are inaccurate, misleading or fabricated yet presented as factual, a phenomenon referred to as “hallucinations.” Notable examples cited include a 2023 incident where a US law firm used ChatGPT to draft a brief and the system fabricated six non-existent rulings, and a separate 2023 incident in which an advertised chatbot produced an incorrect answer that reportedly led to a large stock market loss for its provider. The dossier characterises hallucinations as a widespread challenge for AI operators, with no provider having fully resolved the issue.

Italian law and enforcement responded by assessing whether introducing GenAI services that produce inaccurate content without sufficient transparency could amount to an unfair commercial practice under Articles 20, 21 and 22 of Legislative Decree No. 206/2005 (Consumer Code), because consumers might be misled about the characteristics of the AI system. The Italian Competition Authority concluded three investigations into companies developing chatbots and apps after accusations of insufficient user information. The companies committed to add permanent chat disclaimers (including in Italian) and provide Italian-language terms and conditions in line with Article 9 of the Consumer Code; the Authority found these undertakings sufficient and closed the proceedings without finding infringements. The case is presented as illustrating the link between AI regulatory obligations (citing Article 50 of the AI Act on transparency) and consumer-protection law where informational gaps may be prohibited by the Consumer Code.