Upon its debut in late 2022, ChatGPT brought attention to the revolutionary possibilities of artificial intelligence (AI). The underlying technology of this advanced chatbot represents a significant leap forward in the realm of AI. Unlike traditional approaches that focus on analyzing or categorizing existing data, generative AI has the remarkable ability to craft entirely novel content, ranging from text, images, and audio to synthetic data. This innovation is poised to unleash unprecedented levels of human creativity and productivity across various domains, including business, science, and society as a whole.
Unlocking Potential: Classic NLP vs. Generative AI in Chatbot Design
February 8, 2024
February 8, 2024
Generative AI has established a presence in various industry sectors and is swiftly permeating both commercial and consumer markets. According to McKinsey‘s projections, by the year 2030, approximately 30% of U.S. work hours, currently performed through manual efforts, could witness automation, driven by the rapid advancement of generative AI.
Challenges
Like any emerging technology, generative AI also struggles with its own set of challenges and one of them is the potential of the creation of deepfakes, the generation of biased content, and the manipulation of information. Consequently, human-in-the-loop safeguards become necessary for guiding, monitoring, and validating the generated content. It is essential to comprehend and tackle ethical considerations associated with these practices.
Generative AI also sparks inquiries into the legal ownership of both machine-generated content and the data utilized for training these algorithms. According to Nah et al (2023), during ChatGPT’s development, a substantial volume of personal data was utilized, posing privacy threats. As ChatGPT becomes more prevalent, it integrates into daily life, offering convenience but also capturing extensive personal information. The associated risks involve potential intentional or unintentional exposure of private information to the public. To address this, consulting with legal experts is crucial, and a thorough consideration of potential risks and benefits when utilizing generative AI for creative purposes is advised.
Besides, generative AI also poses challenges to the labor market as it finds applications across various industries, potentially leading to job displacement. Nah et al (2023) mentioned that the evolving division of labor between humans and algorithms may render some jobs obsolete, causing workers to be replaced by automation. Conversely, the implementation of generative AI can also create new job opportunities in different sectors. To remain competitive, reskilling is crucial for individuals to collaborate with AI and develop skills that cannot be easily replaced.
In the realm of conversational AI, the quest for more effective and human-like interactions has led to the exploration of various techniques. Two prominent approaches are Classic NLP, combining information retrieval techniques and semantic search for intent prediction, and Generative AI, employing Large Language Models (LLM) for both intent prediction and response generation. In this blog post, we’ll delve into the strengths and weaknesses of these methods, providing insights into their applications and showcasing some examples.
Classic NLP
Classic NLP leverages the strengths of both classical information retrieval and semantic search to enhance intent prediction in chatbots.
Classical information retrieval technique assigns weights to words based on their frequency in a document relative to their occurrence across all documents. In chatbot intent prediction, this technique helps identify keywords and their relevance to specific user queries. Semantic search, on the other hand, focuses on understanding the meaning behind words and queries. This involves analyzing the context, synonyms, and relationships between words to identify the user’s intent more accurately.
Example:
Consider a user query: “What’s the weather like today in New York City?”
Classical information retrieval would prioritize terms like “weather,” “today,” and “New York City” to predict the intent.
Semantic search, however, would recognize the semantic relationships between words, allowing the chatbot to understand the underlying meaning of the query. For instance, it would comprehend that the user is seeking information about the advantages or positive aspects of utilizing classic NLP within chatbots.
Image: Classic NLP – Semantic Search
The synergy between classical information retrieval and semantic search in classic NLP is particularly beneficial in addressing the limitations of each approach when applied individually. Information retrieval may struggle to capture the nuanced meaning of user queries and could potentially miss the intent if the keywords are not explicitly present in the documents. On the other hand, semantic search, while more contextually aware, might face challenges in accurately predicting user intent if it doesn’t have a robust framework for handling specific keywords. By combining these two approaches, classic NLP aims to provide a more comprehensive and accurate understanding of user intent.
Generative AI with Large Language Models
Generative AI, powered by Large Language Models (LLM) such as OpenAI’s GPT-4, takes a different approach. Instead of merely predicting intent, LLMs have the capability to generate human-like responses by understanding the context of the conversation. LLMs can grasp the context of a conversation, enabling more nuanced responses. And unlike fixed responses in intent prediction, LLMs generate diverse and contextually relevant answers.
Example 1:
User: “Can you recommend a good Italian restaurant near me?”
Generative AI could produce responses like:
“Certainly! How about trying Trattoria del Gusto? It’s known for its authentic Italian cuisine.”
“I’d suggest checking out La Piazza for delicious Italian dishes. It’s just a short drive away.”
Example 2:
User: “How do people play Ultimate Frisbee?”
Generative AI could response in detailed such as:
“Ultimate Frisbee is a team sport played with a flying disc (Frisbee). Here’s a basic overview of how people play Ultimate Frisbee:”
While outlining the Team Composition, Starting Play, Movement and Passing etc.
However, generative AI with large language models also faces its own set of challenges. While it excels in generating contextually relevant responses, there’s a potential risk of producing inaccurate or biased information. The model’s responses are based on patterns it learned during training, and it may inadvertently generate content that is misleading or not factually accurate. Moreover, the sheer generative nature of these models might result in responses that sound plausible but lack proper verification. In critical domains where accuracy is paramount, such as healthcare or legal advice, relying solely on generative AI without rigorous fact-checking mechanisms could lead to misinformation.
Hybrid Generative AI - The Best of Both Worlds
To mitigate these challenges, a hybrid approach can be adopted, combining the strengths of generative AI and classic NLP. Our approach employs classic Natural Language Processing (NLP) techniques as the initial step, enabling it to search for answers within the context of the business domain. This ensures that responses are not random but are specifically tailored to the industry, providing accurate and contextually relevant information. In cases where conventional NLP encounters challenges or uncertainties, our system seamlessly transitions to generative AI as a fallback. Importantly, even in this fallback phase, the system maintains a business-focused context, avoiding generic answers from external sources like Google. This meticulous dual-layered approach guarantees that users consistently receive precise and pertinent responses, elevating the efficacy of interactions and keeping the business at the forefront of innovative solutions in the industry. This integration allows the chatbot to leverage the nuanced understanding of context and intent provided by generative AI, while also benefiting from the precision and accuracy of classic NLP in identifying key information.
The Pros & Cons
Classic NLP
Pros:
- Efficiency: Classic NLP is computationally efficient, making it suitable for real-time applications to quickly identify key terms and predict intent.
- Contextual Understanding: Using semantic search, classic NLP recognizes the semantic relationships between words, allowing the chatbot to understand the underlying meaning of the query.
- Predictability: Classic NLP provides a more predictable output, as they are rule-based and rely on predefined patterns. Users have a higher degree of control over the weighting of specific terms and rules.
Cons:
- Limited Adaptability: Classic NLP may struggle with adapting to new or unseen contexts, as it relies on predefined rules and patterns.
Generative AI with LLM
Pros:
- Adaptability: LLMs can adapt to a wide range of user inputs, providing dynamic and relevant responses. It can adapt to various contexts, allowing for more flexible and adaptive conversational agents.
- Learning from Data: LLMs can be fine-tuned on specific datasets to align with the desired tone and style.
Cons:
- Increased Response Time: The sheer size and complexity of LLMs may lead to longer response times in certain scenarios
- Resource Intensive: Training and deploying large language models require significant computational resources.
- Potential for Unpredictability: The generative nature of LLMs might result in responses that are contextually accurate but unpredictable. Control over the output may be challenging, leading to the potential generation of inappropriate or biased content.
Conclusion
Both Classic NLP and Generative AI have their merits, and the choice between them depends on the specific requirements of a chatbot application. Classic NLP provides quick and predictable intent prediction with fine-tuned control, while Generative AI with LLMs offers contextually rich responses at the expense of potential unpredictability. Striking a balance between speed, control, and adaptability is crucial for building effective and user-friendly conversational AI systems. The future of chatbots may see a fusion of these techniques, leveraging the strengths of each to create more intelligent and responsive conversational agents.
Ready to revolutionize the AiChat’s hybrid generative AI approach? Combining our technology with a user-centric design guarantees a seamless experience and concrete business results. Gain a competitive edge by unleashing the complete capabilities of generative AI within your industry. Schedule a demo today to embark on a transformative journey with us!
AiChat is a leading A.I-Powered Conversational Customer Experience platform designed to help brands automate business processes in customer service, marketing and commerce via popular social messaging apps such as Facebook Messenger, WhatsApp, Instagram and Google Business Messages etc. We are proudly trusted by Bayer, TESCO, Marina Bay Sands, MR D.I.Y, Mondelēz, Petron, Unilever and many other enterprises and SMEs across South East Asia.
