Generative AI (GAI), a new and promising aspect of AI, is able to create content, such as, text, image, music, code, and speech, by learning from and using existing content and data. It is the latest “game changing” buzz. ChatGPT, DALL-E, Jukebox are examples of text, image, and music lyrics generating GAI applications. The content is created via interactions with a chatbot and a Q&A interface. While nascent, this technology is now being tested for applicability in business. The objective of this article is to highlight business risks of generative AI business applications deployments, emphasizing data, and risk mitigation approaches.
There has been a tremendous surge in interest and applications of GAI to business application use cases since ChatGPT was introduced in November 2022. A plethora of start-ups are offering GAI services in various domains and frenzied venture funding is also pouring in. A key question at senior leadership and Board levels is “what is your AI strategy?”. Many organizations, large and small, have either tiptoed into or dove headlong into GAI. Information mis-use lawsuits have also started – a class action lawsuit has been filed against Google over alleged mis-use of user’s personal information during training Bard.
Some business functions exploring Generative AI include:
- Sales and Marketing to create new content.
- Customer service and account management to proactively manage issues, manage conversations, provide analytic insights, session summaries,
- Operations to streamline tasks and proactively identify bottlenecks.
- Supply chains to predict and simulate supply chain visibility across multiple geographies and tiers.
- Logistics and distribution to optimize delivery routes, costs, transmit times, maintain fleet.
- IT – to write code, documents, and create dashboards on-demand.
- Legal- use in case discovery and risk analysis.
How does GAI work: Neural network based Large Language Models (LLM) form the basis of GAI. A deep learning neural network consists of multiple layers of interconnected neural nodes. ChatGPT’s deep learning neural network receives a text string as input and generates a response as output. Text input in chunks, called tokens, is first encoded into numerical data before being fed into the network. The output of the model is a response, one word at a time, with each new word dependent on the previous word. This is based on a probability calculated by the model; that is, the word with highest contextual probability is output. The model is initially trained with supervised learning to recognize data patterns using labeled examples. In the next stages, the model is trained and refined using reward and reinforcement. ChatGPT’s training dataset essentially is the entire available public data and is estimated upwards of 100 trillion data parameters for GPT-4 and upwards of 1.75 billion for GPT3.5.
Google, Microsoft, Open AI, Meta, DeepMind, Nvidia, are some of the leading LLM developers. These models, while in development and test mode, are also being made available for fee to users via API connections. A start-up would use the LLMs to train their business specific application via the LLM API integration.
Known deficiencies and risks: The rise of the increase of LLMs use in daily life and business operation has also exposed the deficiencies of the LLM outputs and risks due to data leakage and poisoning, IP proliferation potential, and system hijacks. Hallucinations- making up answers and confidently presenting false information, automation bias – bias due to repeated training on similar data, jailbreaks – finding devious workarounds to generate harmful output or commands, bias reinforcement -repeating back user preferences, and scalability are some known risks. The willingness of users to trust the tool without knowing its operations or the underlying risks also amplifies this problem.
Risk areas: How different is the use risk of GAI LLM use as compared to other cloud-based applications such as ERP? Firstly, when using SaaS applications, there is no need to share organizational data for model refinement and training. ERP systems are confined and don’t have risk of training related data leakage found in GAI systems. All transactions and sensitive business data are exchanged using current security protocols guarding against human and technology vulnerabilities. Cloud-based applications also don’t encounter the risk of repeated training due to ever-increasing use of various data sources with no data sensitivity guardrails. The GAI dependence on ever-increasing data corpus increases the possibility of LLM memorization data leakage and external hacks. These risks are further exacerbated when cross-functional processes are integrated and GAI used. An example is the integration of GAI with ERP involving various cross-functional usage, such as, customer service, incident management, finance. It is possible that sensitive data can be leaked during interactive Q&A sessions. The following are the major risks from training and data use.
Internal risks: Fine-tuning the model with sensitive org data is also risky as is direct employee input of sensitive data into LLM training models. Samsung reported employee/s inputting confidential information, including confidential and highly sensitive source code, in an AI chatbot attempting to correct coding errors. As there is no user understanding of the internal operations of the LLMs, it is unknown how fine-tuning data is used, stored, and output by the model. Model fine-tuning is devoid of traceability of the answer, it is impossible to control and limit documents and information to certain users or groups, and costs are high.
External hack and attack risks: Data poisoning – intentional corrupting of training data by attacker- risks increase with increasing data source input and variety used for training often from open-sources with minimal oversight. Prompt injections- manipulate prompt inputs to an LLM to generate malicious output- risk remote control of LLMs and exfiltration of data. Since numerous open-source data and other AI applications may be involved, the potential of the scale of hacks is manifold compared to current breaches.
Major Data Risks:
- Training:
- Leakage of data from model memory and memory inference where model predicts the full set of data given a sub-set, for example, credit cards, and healthcare PHI. If the query is to retrieve the last four digits of a credit card, the returned information may include the entire card number. Similarly, when creating patient visit summaries, the output risk is redundant or switched information, such as, notes, learned during training from a large data set of similar cases.
- LLM memorization is the recall of specific training data similar to data leakage except that even if sensitive data is not part of the training input, there may be contextual data in the LLM which is extracted and output.
- Jailbreaks: This are exploits of the model to ask the model to respond as a bad actor and execute behavioral guidance provided by the bad actor. The model can be instructed to role play the exact opposite of the conventional output the GAI is trained for.
- Data extraction: Occurs when chatgpt generated malicious data extraction back-end query attacks are injected to generate harmful data output. This is similar to SQL injections.
- API risks: The use of trojans and prompt injections to complete nefarious activities, such as, impersonate to read emails, compose emails, search user address books, and send emails. Trojans and prompt injections can also allow infiltration to other API integrated LLMs. With scale, this can be seriously problematic as it would be almost impossible to know the source of the compromise.
- Source code hacks where malicious, obfuscated code is inserted in a code block which the developer may or may not check as they trust the engine. Upon executing the code, this will propagate breaches and data exfiltration.
How to mitigate the risks:
- Understand each type of risk discussed above and its potential for breach.
- Identify where to use. These technologies are not deterministic anymore and should cause the stakeholders to think carefully about the impact/cost of the false positives/false negatives. For example, “should I use LLM to process legal documents? Won’t use until proven”; “should I use LLM for an engagement chatbot? Yes”.
- Shine the visibility, fronted by CISO and IT, to planned or deployed projects or applications within the organization. Monitor deployments for success/failure, data and process risks.
- Create active organizational guardrails and policies for GAI application deployments. Decide what infrastructure stack to fit, for example, private cloud vs. public cloud.
- Clearly identify and understand cross-functional data sharing – provide controls or blocking for sensitive, PII or confidential data use. Validate organization data for model fine-tuning against inadvertent release of sensitive data and mask or anonymize all such data.
- Conduct robust security checks and tests before deploying a fine-tuned model. These include deploying available AI-based solutions for detecting system anomalies and bad actor behaviors for prompt injections, data leaks, memorization errors and the others identified above. Also include capabilities to test model output; for example, does a medical summary contain other patient information or deductions? Meaning, sticking memorization specifics of what is learned from all similar cases to one patient’s case summary.
- Develop processes and tools to validate the legitimacy of GAI supplied codes for injections and malware for both internal development and OSS supply chain.
- Use private chatgpt: It is possible to train a private chatgpt with an organization’s own data and inputs. Either a pre-trained model or a ground-up built model and fine-tuning can be used with corporate data. It is recommended to separate the LLM from the corporate knowledge base – an open question when corporate data is required for training. Thus, protocols for data separation are needed. Creating a private chatgpt can be very expensive as it requires technical expertise, large datasets, and resources and will often be unaffordable.
As GAI applications and LLMs are still in its infancy, LLM guardrails and policies are currently insufficient and will evolve with time. It is highly unlikely that the risk mitigation tools for steps outlined above are available; some may be in development. Until AI-based robust risk mitigation tools and policies are developed, available, and installed, organizations should take a very careful approach to its deployment mindful of the enormous risks at stake.