AI development has been largely restricted to a small number of software engineers. This is a form of priesthood where the majority of people rely on tech companies to build AI systems for them. This is not enough to address the vast array of applications that AI could serve. There are two upcoming trends in AI technology that aim to make this more accessible and widespread: data-centric AI and prompt engineering.

Data-centric AI and prompt engineering both offer ways for individuals without extensive AI or programming expertise to benefit from and interact with advanced AI systems. They can be instrumental in upskilling people who don’t have a background in AI, and can enhance their productivity, creativity, and problem-solving capabilities in various professional contexts.

• Data-Centric AI: By focusing more on the quality, relevance, and processing of data, rather than the complexity of the AI models, data-centric AI democratizes the use of AI technologies. It enables domain experts, who may not be AI specialists but have deep understanding of their field and the data it generates, to contribute to AI projects. They can do so by curating and labeling datasets, identifying important features and anomalies, and providing context-specific insights. This way, data-centric AI can help non-AI people to upskill by learning how to work with data, understand its value, and see its impact on AI performance.
• Prompt Engineering: Prompt engineering enables users to interface with AI systems by crafting natural language prompts, which is a significantly lower barrier to entry than coding. For example, using a tool like GPT-3, a person could generate a wide range of content or solve problems simply by crafting the right prompt. This can be used in various fields like content creation, customer support, education, and more. Learning how to effectively use and engineer prompts is a much simpler task than learning to code, but it can still provide a powerful skill set for interacting with AI.

In both cases, individuals can learn to effectively utilize these tools through education and practice, opening up new opportunities for them to leverage AI in their work without needing to become AI experts. This can lead to greater productivity, novel solutions to problems, and innovative applications of AI.

Upskilling in Data-Centric AI and Prompt Engineering

Upskilling in data-centric AI and prompt engineering can be a rewarding journey, even for individuals with no background in AI. There are numerous resources, online courses, and tools available today that can facilitate this learning process. Remember, learning AI is a journey, and it’s okay to start slow and gradually build up your understanding and skills. Don’t get discouraged if things seem confusing at first. With persistence and the right resources, you can certainly upskill in data-centric AI and prompt engineering. For example Andrew Ng’s new course “ChatGPT Prompt Engineering”, created together with OpenAI, is available now for free! Access it here: https://lnkd.in/gmNZYMEy

Here are some steps one can take for upskilling:

• Basic Understanding of AI and Machine Learning: Before diving into data-centric AI or prompt engineering, one needs to have a basic understanding of AI and machine learning. Many online platforms offer beginner-level courses that cover these fundamentals. Websites like Coursera, edX, and Khan Academy have courses in AI and machine learning, often taught by experts in the field.
• Understanding Data: Data-centric AI is all about understanding and manipulating data to improve AI models. Courses in data science, statistics, and data analysis can provide the necessary background. Websites like DataCamp and Kaggle offer hands-on courses and competitions that can help you practice these skills.
• Learning about Data-Centric AI: After getting a handle on data science, the next step is to learn about data-centric AI specifically. This involves understanding how to collect, clean, label, and augment data to improve model performance. There aren’t many courses specifically on this topic as of now, but the principles can be found in more advanced machine learning and deep learning courses.
• Understanding Natural Language Processing (NLP): Prompt engineering heavily relies on NLP, which is a branch of AI that deals with the interaction between computers and humans through language. Online platforms like Coursera and edX offer courses in NLP.
• Learning about Prompt Engineering: As prompt engineering is a relatively new field, there aren’t many dedicated courses available. However, you can learn a lot from the documentation and user guides of language models like GPT-3 by OpenAI.
• Hands-On Practice: Theoretical knowledge needs to be supplemented with practical experience. Open-source platforms like Google Colab can be used to practice coding and experiment with models. Kaggle also offers many datasets and competitions where you can apply what you’ve learned.
• Continuous Learning: AI is a rapidly evolving field. It’s crucial to stay updated with the latest research, techniques, and tools. Following AI researchers and institutions on platforms like Twitter, Medium, and arXiv can help.
• Networking and Community Involvement: Join AI and data science communities online (like on Reddit or Stack Exchange), attend webinars, workshops, and conferences. They can be great places to learn from others’ experiences and get your questions answered.
• Build Projects: Finally, nothing beats learning like building your own projects. It could be something as simple as a chatbot or a text generator. This will give you an idea of the challenges faced in real-world applications and how to overcome them.

Let’s discuss the two topics of Data-Centric AI and Prompt Engineering further.

Data-Centric AI

Data-centric AI is a shift in focus from the traditional approach of developing AI systems, which has been largely model-centric. In a model-centric approach, the primary focus is on improving the machine learning models by tweaking the code and algorithms. The same dataset is usually used to train multiple iterations of these models. This approach, while effective, requires a deep understanding of machine learning algorithms and coding, which can limit the number of people who can effectively participate in AI development.

The data-centric approach, on the other hand, emphasizes the importance of the quality, quantity, and diversity of the data used to train the AI systems. It suggests that improvements in the performance of AI systems can be achieved not just by tweaking the models, but by improving the data that these models learn from. In a data-centric AI approach:

• Data Quality: Ensuring high-quality data is of utmost importance. This includes cleaning the data to remove errors, inconsistencies, and duplicates, as well as labeling the data accurately for supervised learning tasks.
• Data Quantity: The more data the model has to learn from, the better it can understand and generalize. This is particularly important for deep learning models, which tend to perform better with larger datasets.
• Data Diversity: To make AI systems robust and fair, they need to be trained on diverse datasets that represent a wide range of scenarios, demographics, and edge cases. This helps to avoid biases and to ensure that the system performs well for a wide range of users and in a variety of conditions.
• Feature Engineering: This involves identifying and creating informative features from the raw data that can help the model to make accurate predictions.

The data-centric approach can democratize AI as it allows people who may not have advanced knowledge of AI algorithms, but who have domain-specific knowledge and understand the data well, to contribute to the development of AI systems. This approach recognizes that often, the people closest to the data are the ones best equipped to improve its quality or generate new, useful features from it. For example, a medical professional might not know how to code a deep learning model, but they can certainly provide valuable insights on medical data, such as which features are most relevant for diagnosing a disease.

Further, the data-centric approach also involves developing technologies and tools that make it easier for non-experts to work with data. This could include tools for easy data labeling, data cleaning, and feature engineering, among others. For example, in the speech given, the speaker mentioned a tool that allows workers in a clothing factory to easily label images of cloth to train an AI system for defect detection.

In addition, the data-centric approach is not just about using more data, but also about using the right data. It encourages careful thought about what data is needed, how it should be collected and labeled, and how it can be used most effectively. This often involves creating a tight feedback loop between the AI system’s performance and the data collection process, so that the data can be continually improved based on the system’s needs.

While the data-centric AI approach has its advantages, it also comes with its own set of challenges. These include issues around data privacy and security, the need for robust data governance practices, and the challenge of ensuring that data is representative and free of biases. Despite these challenges, the data-centric approach holds great promise for making AI development more accessible and for creating AI systems that are more effective and fair. Let’s delve into some examples of data-centric AI to illustrate the concept further.

• Defect Detection in Manufacturing: This is an example mentioned in the speech. Here, a clothing factory wants to use AI to detect defects in cloth. Instead of requiring the factory workers to develop a complex AI model from scratch, a data-centric approach is used. The workers, who have deep domain knowledge about the cloth and potential defects, annotate images of the cloth. They label areas where there are tears, discolorations, or other defects. The model then learns from this carefully curated and labeled data, becoming a tool that can automatically detect defects in the cloth. The success of the model depends more on the quality of the data and less on the complexity of the AI model.
• Medical Diagnosis: In a medical setting, a data-centric approach could involve doctors and medical professionals labeling medical images (like X-rays or MRI scans) to indicate the presence or absence of certain conditions. This labeled data then serves as the foundation for training an AI model to identify these conditions. The experts’ domain knowledge is vital for ensuring the accuracy of the data, and therefore the effectiveness of the resulting AI system.
• Customer Support: Many companies use AI chatbots to handle common customer inquiries. A data-centric approach to developing these chatbots would involve collecting and curating a dataset of past customer interactions, including both the customer’s questions and the support agent’s responses. The chatbot can then be trained on this dataset to handle similar inquiries in the future. The quality and diversity of the collected interactions can have a significant impact on the chatbot’s ability to understand and respond to a wide range of customer questions.
• Autonomous Vehicles: Autonomous vehicles rely heavily on data-centric AI. They need vast amounts of annotated data, including images, sensor data, and lidar scans, to understand and navigate their environment. The process of collecting and annotating this data involves both automated systems and human annotators. The quality and diversity of the data is crucial to the safe and effective operation of the vehicle.

In each of these examples, the emphasis is on the data: its quality, its labeling, and its representation of the problem space. The AI algorithms used are often standard, open-source models, but the effectiveness of the final system hinges on the careful curation and management of the data.

Prompt Engineering

Prompt engineering is a technique used to guide the behavior of AI models, especially those based on large language models like GPT-3 developed by OpenAI. The goal is to provide the model with a starting point, or “prompt”, that helps direct the model’s output in a way that’s useful for the task at hand.

Prompts can be simple or complex, depending on the task. For example, a simple prompt could be “Translate the following English text to French:”, followed by the text to be translated. A more complex prompt might involve a multi-step instruction, or an instruction that requires the model to generate a creative story, poem, or piece of art.

Prompt engineering is a form of “weak supervision” in machine learning, where models are given high-level guidance instead of detailed, instance-by-instance labels. It’s a crucial part of using models like GPT-3 effectively, because these models are “pre-trained” on a large corpus of text, and then “fine-tuned” on a smaller, task-specific dataset. Here are a few examples of how prompt engineering can be utilized:

1.Content Generation

A content creator could use a prompt like “Write a blog post about the health benefits of yoga for seniors” to generate a detailed article. They can further refine the prompt to get more specific content, like “Write an introduction for a blog post about the health benefits of gentle yoga for seniors with arthritis.” Prompt engineering involves creating a precise and detailed command that guides the AI to generate the desired content. This can range from blog posts, marketing copy, and social media updates, to scripts for videos, speeches, or even writing prompts for books. The AI uses the input prompt to create text that matches the given instructions. Let’s expand further on the example, which was about creating a blog post on the health benefits of yoga for seniors:

• Initial Prompt: “Write a blog post about the health benefits of yoga for seniors.” – This is a basic prompt which might lead the AI to generate a general overview of the topic.
• Refined Prompt: “Write an informative and engaging 800-word blog post suitable for seniors, detailing the top five health benefits of regular yoga practice, including improved flexibility, better balance, increased strength, stress relief, and enhanced cardiovascular health.” – This more detailed prompt provides a clearer framework for the AI, specifying the tone (informative and engaging), the target audience (seniors), the word count (800 words), the structure (top five benefits), and the specific benefits to cover.
• Even More Specific Prompt: “Begin an 800-word blog post suitable for seniors with an engaging introduction that draws the reader in and then transitions into a discussion on the top five health benefits of regular yoga practice, including improved flexibility, better balance, increased strength, stress relief, and enhanced cardiovascular health. Include real-life examples and easy yoga exercises they can try at home.” – This prompt is even more specific, guiding the AI to not only discuss the health benefits but also include real-life examples and suggesting some simple yoga exercises.

The key to effective content generation through prompt engineering is to provide as much detail and context as possible. This allows the AI to generate content that closely aligns with your requirements. However, keep in mind that AI-generated content might need editing or proofreading for coherence, accuracy, and alignment with your brand voice.

2. Question Answering

Question Answering (QA) systems are AI models trained to provide direct answers to user queries. They’re often employed in customer service chatbots, virtual assistants, and search engines. With the help of prompt engineering, these systems can provide more accurate, detailed, and contextually relevant responses. The process of prompt engineering in QA involves framing the question in a way that makes it clear what information the AI should be seeking to provide. Here are a few examples:

• Basic Prompt: “What is the weather like?” – This is a simple prompt but it lacks specificity. The AI model may not know where or when to provide the weather information for.
• Refined Prompt: “What is the current weather in New York City?” – This prompt is more specific, instructing the AI to look for real-time weather data for a particular location.
• Detailed Prompt: “What is the current temperature, humidity, and chance of rain in New York City?” – This prompt is even more specific, asking the AI for several different pieces of information about the weather in a particular location.

In a real-world application, the user provides the query (e.g., “What’s the weather like in New York City today?”), and the AI system, based on its training, understands the question and retrieves the relevant information.

Prompt engineering also comes into play when designing the AI’s responses. For instance, instead of a barebones answer like “70°F, 60% humidity, 10% chance of rain,” the AI could be guided to respond in a more human-like, conversational manner: “Today in New York City, it’s currently 70 degrees Fahrenheit with a humidity level of 60%. There’s a 10% chance of rain, so you might not need that umbrella after all!”

The key with prompt engineering in QA systems is to ensure the AI comprehends the nuances of the questions and provides relevant, detailed responses. The prompts not only guide the AI in understanding the question but can also help shape the style and depth of the answers.

3. Creative Writing

Creative Writing refers to the ability of an AI model to generate original, imaginative pieces of text. This could range from poetry and short stories to advertisements and slogans. Prompt engineering plays a crucial role in guiding the AI to create relevant, compelling, and stylistically appropriate content.

Here are a few examples:

• Basic Prompt: “Write a poem.” – This is a simple prompt, but it’s also very broad. The AI has no guidance on the theme, style, length, or tone of the poem.
• Refined Prompt: “Write a haiku about spring.” – This prompt is more specific, providing the AI with the type of poem (haiku) and the theme (spring). This will guide the AI to generate a three-line poem with the traditional 5-7-5 syllable count, focusing on the theme of spring.
• Detailed Prompt: “Write a sonnet in the style of Shakespeare about unrequited love.” – This is a very specific prompt, instructing the AI to generate a 14-line sonnet, using the iambic pentameter and rhyming schemes associated with Shakespeare, and focusing on the theme of unrequited love.

Let’s look at a practical scenario, like creating an advertising slogan for a new eco-friendly shoe brand:

Prompt: “Generate a catchy slogan for an eco-friendly shoe brand that emphasizes comfort, durability, and care for the environment.”

The AI, trained on diverse datasets and understanding the nuances of the prompt, might generate a slogan like: “Step into the future. Comfort, longevity, and the planet at your feet.”

Prompt engineering allows non-AI experts to harness the AI’s creativity and generate text that is contextually relevant, engaging, and tailored to their specific needs. It provides a valuable tool for anyone in a creative field, like writers, advertisers, marketers, or even individuals who want to create unique, personalized content.

4. Coding

Prompt engineering can also be used to guide AI in generating code. This is a significant development because it democratizes access to coding, making it available to non-experts, and streamlines the coding process, potentially reducing errors and saving time.

Here are a few examples of how prompt engineering can be used in coding:

• Basic Prompt: “Write a Python function.” – This prompt is quite vague and does not give the AI any information about what the function should do.
• Refined Prompt: “Write a Python function to add two numbers.” – This prompt provides specific details about what the function should do, enabling the AI to generate code that takes two numbers as inputs and returns their sum.
• Detailed Prompt: “Write a Python function to sort a list of numbers in descending order without using any built-in sort functions.” – This prompt is very specific, instructing the AI to write a function that sorts a list of numbers in descending order, and adds an additional constraint of not using any built-in sort functions.

Let’s consider a practical scenario where a user needs to generate a function for a specific task in JavaScript:

Prompt: “Write a JavaScript function that takes an array of integers and returns the sum of the integers.”

The AI, understanding the requirements from the prompt, might generate a function like this:

This function iterates over the elements in the array, adding each one to a running total, which it then returns.

Prompt engineering in coding can serve as a valuable tool for developers and non-developers alike. It can help with quick prototyping, learning coding syntax and constructs, debugging, and generating code snippets for specific tasks. It’s an excellent example of how AI can augment human creativity and productivity.

5. Translation

Prompt engineering can be very useful for tasks related to language translation. Given the capability of language models like GPT-3 in understanding and generating text in multiple languages, well-crafted prompts can guide the AI to perform complex translation tasks.

Here are a few examples of how prompt engineering can be used in translation:

• Basic Prompt: “Translate this.” – This prompt lacks clarity as it does not specify what text needs to be translated and into what language.
• Refined Prompt: “Translate ‘Hello, how are you?’ into Spanish.” – This prompt provides clear instructions, specifying both the text to be translated and the target language. The AI, guided by the prompt, will generate the Spanish translation: “Hola, ¿cómo estás?”
• Detailed Prompt: “Translate the following English idiom into French: ‘It’s raining cats and dogs.'” – This prompt is more challenging as it involves translating an idiom, which cannot be translated directly. The AI, understanding the semantics of the idiom, would generate the French equivalent idiom for heavy rain: “Il pleut des cordes.”
• Contextual Prompt: “Imagine you’re a professional translator. Translate the following business email from English to Japanese: ‘Dear Mr. Tanaka, I’m looking forward to our meeting next week. Best regards, John.'” – This prompt adds an additional layer of context, implying a certain level of formality and politeness appropriate for a business email. The AI, understanding the context, will generate a formal, polite Japanese translation.

Prompt engineering in translation can be a powerful tool, not only for translating text but also for learning languages, understanding cultural contexts and nuances, and even translating abstract or creative concepts across languages. It’s another way that AI can help break down language barriers and facilitate communication across different cultures and regions.

6. Simulated Conversation

Prompt engineering can be particularly effective in simulating conversations. It can allow an AI to generate conversational responses, or to maintain a conversational thread over multiple turns.

Let’s take a look at how prompt engineering can be used to simulate conversation:

• Basic Prompt: “Respond to this.” – This prompt is vague and doesn’t provide any conversational context, resulting in a likely nonsensical response from the AI.
• Refined Prompt: “Respond as if you are a customer service representative and someone just said, ‘I can’t log into my account.'” – This prompt provides a role and a specific situation for the AI. The AI could then generate a response like, “I’m sorry to hear you’re having trouble logging in. Let’s try to resolve this. Can you please tell me if you’re receiving any error messages?”
• Detailed Prompt: “As a medical professional, respond to a patient who says, ‘I’ve been feeling very tired lately and can’t seem to focus on my work.'” – This more detailed prompt includes a specific role and a more complex scenario. The AI could respond with something like, “I’m sorry to hear you’re feeling this way. It’s important to consider various factors like sleep quality, diet, stress, and physical activity. However, to get a better understanding, I recommend you see a healthcare provider for a thorough evaluation.”
• Contextual Prompt: “Imagine you’re a Shakespearean character. Respond to another character saying, ‘Thou art a villain!'” – This prompt asks the AI to generate a response in a specific style and context. The AI might respond in kind with, “Thou dost wound me with thy words, yet I bear no ill will. For I know in my heart that I am true.”