GenAI is a term that refers to the use of large language models, such as GPT-4, to generate natural language content for various purposes. GenAI projects are becoming more popular and promising, as they can potentially offer solutions for many domains and tasks.
But GenAI projects are not as straightforward as they may seem. They come with many challenges and risks. In my previous blog post, I explained 5 common pitfalls that slow down the GenAI product delivery. In this blog post, I will explore this topic in more depth and suggest some strategies to smooth the journey.
Unrealistic Expectations: Bridging the Gap Between Technical and Non-Technical Stakeholders
One common challenge in GenAI projects is the misalignment of expectations between technical teams and non-technical stakeholders. Often stakeholders involved in GenAI projects get their expectations based on the fast prototypes in hackathons. Although hackathons are fun and a great way to learn, they often show only the feasibility of solving a specific use case. Here a problem arises, some of the stakeholders are under the impression that just because something is feasible does not mean it is suitable, reliable, and even the best way to solve that problem.
The lack of a deep understanding of the complexities involved in working with large language models makes non-technical stakeholders overestimate the capabilities of GenAI technologies and underestimate the effort required for implementation. For instance, they might assume that because a large language model can generate textual output, it can seamlessly address any problem without considering factors like output quality, reliability, or deployment intricacies.
This disconnect can result in frustration and wasted resources when projects fail to meet inflated expectations.
Strategies for Aligning Goals and Understanding Complexities
To avoid this pitfall, here are some strategies that can help:
- Educate and communicate with the stakeholders: Foster an open dialogue to explain the nuances of GenAI projects. Provide insights into how these models operate, their limitations, and the resources necessary for success.
- Establish realistic goals: Collaborate with stakeholders to define achievable project objectives, including scope, timeline, and budget. Regularly update them on progress and address any concerns promptly.
By fostering understanding and collaboration, teams can navigate GenAI projects more effectively, leading to better outcomes and stakeholder satisfaction.
The Science of GenAI: Mitigating Risks Through Rigorous Evaluation and Testing
Another common challenge in GenAI projects stems from the accessibility of GenAI models, which attracts individuals from diverse backgrounds to the field. However, this accessibility sometimes leads to the overlooking of traditional AI approaches in favor of large language models (LLMs), even when they may not be the most suitable solution. They may not compare the LLM with other methods or baselines, to see how it differs or improves. For instance, attempting to solve problems with GenAI that could be addressed more effectively using other methods, such as LightGBM for time series analysis, can result in unnecessary complexity and suboptimal outcomes.
Furthermore, many individuals entering the GenAI space lack a scientific approach to developing and evaluating AI products iteratively. Without rigorous testing, experimentation, and validation, the reliability and effectiveness of these projects may be compromised. They may not test the LLM on different datasets, scenarios, or domains, to see how it performs and generalizes. They may not evaluate the LLM on various dimensions, such as fluency, coherence, relevance, accuracy, diversity, or ethics. They may rely on intuition, anecdotal evidence, or subjective judgment, rather than empirical data, experiments, or metrics. This can lead to poor quality, unreliable, or harmful output, that may not meet the needs or expectations of the end-users.
Ensuring Reliability, Effectiveness, and Ethical Deployment
To avoid this pitfall, here are some strategies that can help:
- Use the right tool for the right problem: Before choosing an LLM for your project, consider if it is the best fit for your problem. Compare it with other methods or baselines, and evaluate its strengths and weaknesses. Choose the method that offers the best trade-off between performance, complexity, and cost.
- Follow a scientific process: Develop and evaluate your GenAI product using a systematic and iterative process. Test your LLM on different datasets, scenarios, or domains, and measure its performance on various dimensions. Use data, experiments, and metrics to guide your decisions and improvements. Validate your results with external feedback and user testing and always ask the hard questions:
- Is the large language model the best or the only method for your task or domain, or are there other alternatives that may be more suitable or efficient?
- How can you ensure that the output of the large language model is not only fluent and coherent but also relevant and accurate for your task or domain?
- How can you verify and validate the output of the large language model, not only once, but multiple times, using different methods and metrics?
- How can you handle and correct the errors, inconsistencies, or biases that may occur in the output of the large language model?
By asking and answering these questions, you can improve the quality and reliability of the output of the GenAI project, and avoid the pitfalls of relying on intuition, anecdotal evidence, or subjective judgment.
From Chaos to Clarity: Crafting a Strategic Vision for GenAI Projects:
A third pitfall that I see is that some companies do not have a clear AI strategy or focus, and their GenAI projects are too broad and vague. They are still in a chaotic state of trying and experimenting, without a clear vision or direction.
For example, they may not have a clear understanding of the business problem, the customer need, or the value proposition of their GenAI project. They may not have a clear definition of the scope, the target audience, or the success criteria of their GenAI project. They may not have a clear plan, roadmap, or timeline for their GenAI project.
This can lead to not spending enough time to investigate the ideas deeply enough and make sure that the ideas are valid, reliable, and reproducible. This is one of the reasons that many GenAI projects end up being abandoned, delayed, or ineffective.
Defining Scope, Audience, and Success Criteria for Sustainable Progress
To avoid this pitfall, here are some strategies that can help:
- Define and align the AI strategy and focus for the GenAI project. Have a clear and shared vision and direction for the GenAI project, and ensure that it aligns with the business goals, needs, and values of the company and the customer.
- Have a clear and specific scope, target audience, and success criteria for the GenAI project. Ensure that they are measurable and achievable. Have a clear and realistic plan, roadmap, and timeline for the GenAI project, and ensure that they are feasible and flexible.
- Spend enough time to investigate and validate the ideas for the GenAI project. Conduct thorough research, analysis, and testing to ensure that the ideas are sound, reliable, and reproducible. Seek feedback and input from experts, stakeholders, and end-users.
By doing this, the GenAI project will have a clear purpose and direction and will be more likely to deliver valuable and beneficial outcomes.
Conclusion
GenAI projects are exciting and innovative, but they also come with challenges and risks. To ensure the success and quality of GenAI projects, it is essential to avoid these three common pitfalls: having unrealistic expectations from non-technical people, lacking a scientific approach and validation, and missing a clear AI strategy and focus. By doing this, GenAI projects can offer valuable and beneficial solutions for various domains and tasks.
