A recent evaluation by LatticeFlow indicates that several major artificial intelligence models from companies including Meta and OpenAI are struggling with compliance to the forthcoming EU AI Act, particularly in terms of cybersecurity and discriminatory outputs. While models generally scored well, specific shortcomings were identified that may necessitate companies to allocate resources to meet regulations, with significant penalties imposed for non-compliance. The findings spotlight the ongoing challenge of integrating ethics and security into AI technology as European regulators develop enforcement measures for the AI Act.
Recent assessments reveal that several leading artificial intelligence models do not fully comply with critical European regulations, particularly in the areas of cybersecurity and bias in output. This information, acquired by Reuters, has surfaced amidst ongoing discussions regarding the European Union’s (EU) new AI regulations, which gained momentum following the public introduction of ChatGPT by OpenAI in late 2022. In response to the rapid uptake of general-purpose AI technologies, European legislators initiated the formulation of specific guidelines. A newly developed compliance verification tool, known as the “Large Language Model (LLM) Checker,” has been created by Swiss startup LatticeFlow in partnership with EU officials and supported by researchers from ETH Zurich and INSAIT. This tool conducted evaluations on generative AI models from major technology firms such as Meta and OpenAI, assessing them against the forthcoming requirements of the EU AI Act, set to be implemented progressively over the next two years. The models tested received scores ranging from 0 to 1, and the results indicate that models by companies such as Alibaba, Anthropic, OpenAI, Meta, and Mistral performed relatively well, generally achieving average scores of 0.75 or higher. However, the findings from LatticeFlow’s LLM Checker illuminated specific compliance shortcomings that may necessitate resource reallocation by these companies to adhere to EU regulations. Noncompliance with the AI Act could result in penalties of up to $38 million or 7% of a company’s global annual revenue. Although the EU continues to refine how the AI Act will be applied to generative AI technologies, LatticeFlow’s findings provide an important preliminary understanding of the potential areas where these companies may struggle with compliance. Notably, issues regarding discriminatory outputs, reflecting ingrained human biases across various demographics, persist in the development of generative AI models. For instance, OpenAI’s “GPT-3.5 Turbo” scored a mere 0.46 in this context, while Alibaba’s model received a score of 0.37, indicative of ongoing challenges in mitigating such biases. When assessing the vulnerability of models to security threats, particularly prompt hijacking – wherein attackers mask malicious prompts as legitimate to gain unauthorized access to sensitive information – Meta’s “Llama 2 13B Chat” earned a score of 0.42, and Mistral’s model received a 0.38. In contrast, the “Claude 3 Opus” model from Anthropic achieved a commendable average score of 0.89, denoting a stronger alignment with the regulatory expectations. LatticeFlow’s tool is slated to evolve alongside the ongoing introduction of further enforcement measures. The CEO and cofounder Petar Tsankov highlighted the overall positive nature of the assessment results, emphasizing that they provide a conceptual framework for companies to enhance their models in accordance with the AI Act’s stipulations. “With a greater focus on optimizing for compliance, we believe model providers can be well-prepared to meet regulatory requirements,” Tsankov stated. While the European Commission is unable to authenticate external assessment tools, it has acknowledged the study and the evaluation platform as a laudable initial effort in translating the EU AI Act into actionable technical standards. An official spokesperson for the Commission remarked, “The Commission welcomes this study and AI model evaluation platform as a first step in translating the EU AI Act into technical requirements.”
The European Union is in the process of establishing comprehensive regulations for artificial intelligence (AI) technologies, particularly in light of the rapid advancements and popularization of models like ChatGPT. The introduction of the EU AI Act aims to create safety and compliance benchmarks that AI companies must meet to ensure ethical and non-discriminatory use of AI systems. Following the release of ChatGPT, which garnered significant public attention, EU lawmakers were prompted to expedite the drafting of regulations to mitigate potential risks associated with generative AI. In response, various tools have been developed to assess compliance with these forthcoming regulations, leading to an increased focus on identifying areas of concern, such as cybersecurity resilience and biases in AI outputs.
In summary, the introduction of the LLM Checker by LatticeFlow has uncovered both strengths and weaknesses in the compliance of major AI models with the EU’s upcoming AI Act. While some models achieved high scores, significant challenges remain, particularly concerning discriminatory outputs and cybersecurity risks. As the EU works towards finalizing the enforcement of the AI Act, companies are urged to proactively address the identified compliance gaps to avoid substantial penalties. The study serves as a catalyst for improving AI model safety and adherence to emerging regulatory standards, marking a significant step toward responsible AI development in Europe.
Original Source: www.voanews.com
Leave a Reply