As a JavaScript developer, you've probably seen Express controllers wrapped in try-catch blocks to handle async errors. While this works, it can make your code repetitive and cluttered. Enter express-async-handler: a simple tool that eliminates the need for try-catch in every controller. It's designed to handle async errors keeping your code clean and easy to read. In this post, we'll explore how express-async-handler can simplify your Express code and make error handling a breeze.

The Current Scene: Try-Catch Overload

Picture this in your Express app:

export const getUserById = async (req, res) => {
  try {
    const user = await User.findById(req.params.id);
    if (!user) return res.status(404).json({ message: "User not found" });
    res.status(200).json(user);
  } catch (error) {
    res.status(500).json({ message: "Server error", error: error.message });
  }
};

// Using the controller
app.get("/users/:id", getUserById);

See that try-catch wrapping everything? It's like a safety blanket for your code. It works, sure, but imagine doing this for every single controller.

Enter express-async-handler: Your Code's New Best Friend

1. Install it:

npm install express-async-handler

2. Refactor your controller by removing try-catch:

export const getUserById = async (req, res) => {
  const user = await User.findById(req.params.id);
  if (!user) return res.status(404).json({ message: "User not found" });
  return res.status(200).json(user);
};

3. Use it in your route:

import asyncHandler from "express-async-handler";

app.get("/users/:id", asyncHandler(getUserById));

4. Now that we've streamlined our controllers, let's add a global error handler to catch any errors:

import express from 'express';

const app = express();

// Global Error Handler
app.use((error, req, res, next) => {
  console.log.(`Oops! ${error.stack}`);

  res.status(500).json({
    status: false,
    statusCode: 500,
    message: `Something went wrong on our end. We're on it!`
  });
});

That's it! No more try-catch clutter. If an error pops up, express-async-handler catches it and passes it to Express's global error handler. With express-async-handler, you're writing less boilerplate and focusing more on what your code actually does.

Middleware: Your Request's Pit Stop

Middleware is a function that runs between receiving a request and sending a response. Think of it as an assistant that can:

• Check if a user is logged in

• Validate data

• Log request details

• Modify the request or response

Here’s a basic middleware example:

const logRequest = (req, res, next) => {
  console.log(`Received a ${req.method} request to ${req.url}`);
  next(); // Don't forget this! It passes control to the next middleware
};

app.use(logRequest); // Use this middleware globally for all routes

In this case, logRequest logs all incoming requests and is used globally, meaning it will run on every route in your app.

How Middleware Works

Middleware functions take three arguments: req (request), res (response), and next. The next() function is crucial—it passes control to the next middleware in the stack. Without it, the request-response cycle would stop and the request would hang.

Middleware can be applied globally, as in the logRequest example, or individually to specific routes. For instance:

router.post(
  '/',
  validateRequestBody(todosSchema), // Middleware for request validation
  asyncHandler(todosController.createTodo)
);

Here, validateRequestBody checks whether the request body matches the todosSchema before the request reaches the controller. In a similar fashion, you can add custom middlewares for tasks like validating data, authenticating users, or logging specific details on individual routes or across your entire app. Our controllers function as middleware too.

Global Error Handler: Your App's Safety Net

A global error handler in Express is special middleware that catches errors from anywhere in your app. Here’s a simple example:

app.use((error, req, res, next) => {
  console.error(error.stack);
  res.status(500).send('Oops! Something went wrong');
});

In this basic setup, any unhandled errors will be logged, and the user will get a general error message. But in a real-world app, you often need more specific error handling based on the error type.

The great thing about this approach is that it works hand-in-hand with express-async-middleware and other custom middlewares like validateRequestBody that we discussed earlier:

router.post(
  '/',
  validateRequestBody(todosSchema),
  asyncHandler(todosController.createTodo)
);

app.use((error, req, res, next) => {
  logger.fatal(`Error: ${error.stack}`);

  if (error instanceof ValidationError) {
    return res.status(400).json({
      message: 'Validation failed',
      errors: error.validationErrors
    });
  }

  if (error.code === 'FOREIGN_KEY_CONSTRAINT') {
    return res.status(500).json({
      message: 'Database error: Constraint violation. Please ensure that related data exists before proceeding.'
    });
  }

  return res.status(500).json({
    message: 'Something unusual happened'
  });
});

If validateRequestBody throws a validation error, it’s automatically caught by the global error handler. Similarly, database-related errors can also be managed here.

Using this centralized error handling along with express-async-handler and other custom middlewares means you no longer need repetitive try-catch blocks in each route. Instead, you have a streamlined, clean solution that handles errors in a single place, making your app more maintainable and easier to debug.

Cracking Open express-async-handler

If you check the express-async-handler codebase, it's just 8 lines of code—yes, only 8! It’s a simple curried function—a function that returns another function. Below is a simplified version, let's peek under the hood:

export function asyncHandler(controllerFunction) {
  return function asyncMiddleware(req, res, next) {
    const controllerPromise = controllerFunction(req, res, next);
    return Promise.resolve(controllerPromise).catch(next);
  };
}

1. The Wrapper: asyncHandler is the main function that creates a middleware wrapper. It takes a controllerFunction as its argument, which is the function that handles the route logic.

2. The Return: The asyncHandler function returns another function (asyncMiddleware). This returned function is the actual middleware used in Express.

3. The Magic: When this function runs, it:

   • Calls your controller function

   • Wraps the result in a Promise (in case it's not already one)

   • Attaches a .catch() handler to the promise. If the promise is rejected (an error occurs), the error is passed to the next function, which forwards it to the global error handler.

In plain English: express-async-handler takes your controller, runs it, and makes sure any errors are caught and sent to the global error handler. This tiny piece of code saves you from writing try-catch blocks everywhere. It's a small tool that makes a big difference in keeping your code clean and error-free.

When It’s Okay to Use try-catch in Controllers

There’s nothing wrong with using a try-catch block inside a controller when you need to handle specific, localized errors. The key takeaway is balance. For global or repetitive errors, using express-async-handler and a global error handler middleware is the best approach. However, for unique situations that are controller-specific, a try-catch makes sense and is perfectly valid:

export const getUserById = async (req, res) => {
  try {
    const user = await User.findById(req.params.id);
    if (!user) return res.status(404).json({ message: "User not found" });
    return res.status(200).json(user);
  } catch (error) {
    // Handle specific route errors, e.g., database constraints
    if (error.code === "E203") {
      return res.status(500).json({ message: "Server error", error: error.message });
    }

    // Let the global error handler take care of any other errors
    throw error;
  }
};

app.get("/users/:id", asyncHandler(getUserById));

Wrapping Up: Cleaner, Safer Express

We've explored how to simplify error handling in Express, and here's the takeaway:

• Say Goodbye to Try-Catch Overload: express-async-handler eliminates repetitive try-catch blocks.

• Cleaner Code: Your controllers are now focused and streamlined.

• Global Error Handling: Catch errors with a centralized error handler.

• Middleware Magic: Middleware transforms your app's flow and keeps things tidy.

• Simple Yet Powerful: A few clever lines of code make a big difference.

If you’re looking to build Express apps from scratch, check out this complete TypeScript starter that follows clean coding principles and includes end-to-end testing: express-typescript-starter. I’ve also written a 7-part series on it—take a look here. Until next time Happy coding!

In the previous post, we built a simple pizza chatbot using the OpenAI SDK that could handle pizza orders. We discussed several features necessary for a consistent chatbot experience, such as streaming LLM responses, parsing markdown content generated by the LLM into HTML, and exploring alternative LLMs and SDKs for potentially more cost-effective options.

In this second part on building a pizza chatbot with Node.js, we'll implement these features. If you haven’t read the first part yet, I recommend doing so before diving into this post.

Streaming OpenAI Responses in an Express API

To enable streaming responses in our Express API, we need to pass the stream: true parameter to the OpenAI SDK. Instead of waiting for the LLM to generate the entire response, this approach streams the data as it arrives, enhancing the chat experience—especially for longer responses. This is the same technique used in ChatGPT-like interfaces to provide a smoother and more interactive user experience.

app.post("/api/chat-streaming", async (req, res) => {
  const conversations = req.body.conversations;

  try {
    const response = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [...context, ...conversations],
      stream: true,
    });

    const stream = response.toReadableStream();
    const reader = stream.getReader();

    const decoder = new TextDecoder();
    let assistantMessage = "";

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      const chunk = decoder.decode(value);
      const lines = chunk.split("\n");

      for (const line of lines) {
        if (line.trim() === "") continue;
        const json = JSON.parse(line);
        const content = json.choices[0]?.delta?.content || "";
        assistantMessage += content;
        res.write(content);
      }
    }

    return res.end();
  } catch (error) {
    console.error("Error during streaming:", error);
    return res.status(500).json({ error: "An error occurred during streaming" });
  }
});

The streaming part involves getting a continuous flow of data from OpenAI's API and sending it to the client in real-time. Here’s how it works:

• response.toReadableStream():

  • The response object from the OpenAI API call has a method called toReadableStream(). This converts the response into a stream that can be read piece by piece instead of all at once.

  • Think of a stream like a faucet where water (data) flows gradually, instead of filling a whole bucket and then using it. You get the data as soon as it arrives.

• const reader = stream.getReader():

  • The stream is read using a reader. The getReader() method gives us a way to read the stream's data chunk by chunk.

• const decoder = new TextDecoder();:

  • The TextDecoder is used to convert the chunks of data (which are in a raw format) into readable text.

• Reading the Stream in a Loop:

  • The while (true) loop continuously reads from the stream until there is no more data (done becomes true).

  • await reader.read() reads the next chunk of data from the stream.

    • done: A flag that tells if the stream has finished (true when there’s no more data).

    • value: The actual chunk of data.

• Processing the Data:

  • Each chunk of data is decoded into text using decoder.decode(value).

  • The decoded text (which might be several lines) is split into individual lines using chunk.split("\n").

  • The loop goes through each line. If a line is empty (just a newline character), it skips it.

• Sending Data to the Client:

  • For each non-empty line, the line is parsed as JSON (JSON.parse(line)), and the relevant content (the AI's message) is extracted.

  • res.write(content) sends this content to the client as soon as it’s available.

  • This means the client starts receiving parts of the AI's response immediately, without waiting for the whole response to be ready.

• Ending the Response:

  • Once the stream has no more data (done becomes true), the loop exits, and res.end() is called to signal that the response is complete.

Receiving Streams on the Frontend

Now that we have the backend code set up for streaming, let’s look at how to handle this data on the frontend. Replace the script code from the previous tutorial with the following:

<script>
  document.addEventListener("DOMContentLoaded", () => {
    const conversations = [];

    const initialMessage =
      "Hi there! 😊 Ready to place your pizza order 🍕 or have any questions? Let me know!";
    addMessageToChat("Assistant", initialMessage, "assistant");

    document
      .getElementById("chat-form")
      .addEventListener("submit", async (event) => {
        event.preventDefault();

        const userMessage = document.getElementById("message-input").value;
        if (!userMessage) return;

        addMessageToChat("User", userMessage, "user");
        conversations.push({ role: "user", content: userMessage });
        document.getElementById("message-input").value = "";
        const response = await fetch("/api/chat-streaming", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ conversations }),
        });

        const reader = response.body.getReader();
        const decoder = new TextDecoder();
        let assistantMessage = "";

        // Create the assistant message element once
        const assistantMessageElement = addMessageToChat(
          "Assistant",
          "",
          "assistant"
        );

        while (true) {
          const { done, value } = await reader.read();
          if (done) break;

          const chunk = decoder.decode(value);
          assistantMessage += chunk;

          // Update the existing message element with the latest content
          updateMessageInChat(assistantMessageElement, assistantMessage);
        }

        conversations.push({
          role: "assistant",
          content: assistantMessage,
        });
      });

    function addMessageToChat(sender, message, className) {
      const chat = document.getElementById("chat-area");
      const messageElement = document.createElement("div");
      messageElement.className = `chat-message ${className}`;
      messageElement.innerHTML = `<strong>${sender}:</strong> <span class="message-content">${message}</span>`;
      chat.appendChild(messageElement);
      chat.scrollTop = chat.scrollHeight;
      return messageElement.querySelector(".message-content"); // Return the span where the message content is displayed
    }

    function updateMessageInChat(element, message) {
      element.innerHTML = message;
      const chat = document.getElementById("chat-area");
      chat.scrollTop = chat.scrollHeight; // Keep the chat scrolled to the bottom
    }
  });
</script>

Let's break down the streaming part of our frontend code, focusing on how messages from the backend are received and displayed in real-time.

1. Sending the User Message and Making the POST Request:

   • When the user submits a message in the chat, it’s added to the conversations array and displayed in the chat window.

   • A POST request is then sent to the /api/chat-streaming endpoint with the conversations array in the request body.

2. Receiving the Stream:

   • Once the POST request is sent, the frontend starts receiving a streamed response from the server.

   • The response.body.getReader() method is used to create a reader that reads the incoming stream of data chunk by chunk.

3. Decoding the Stream:

   • A TextDecoder is used to convert each chunk of raw data into a readable text string.

4. Creating the Assistant Message Element:

   • When the streaming starts, an empty message element is created for the assistant using the addMessageToChat function.

   • This element is added to the chat area, but it starts with no content because the full response hasn’t been received yet.

   • The addMessageToChat function returns the specific span element where the assistant's message content will be displayed.

5. Updating the Assistant Message in Real-Time:

   • As chunks of the assistant's message are received from the stream, the message content is gradually built up in the assistantMessage variable.

   • After each chunk is received, the updateMessageInChat function is called to update the content of the already-created message element in the chat.

   • This means that the assistant’s message appears to be typing out in real-time, giving the user immediate feedback as the AI generates its response.

6. Finalizing the Message:

   • Once the stream is fully read (indicated by done being true), the full message from the assistant is finalized and added to the conversations array.

Parsing Markdown to HTML on the Frontend

One issue we’ve encountered is that while the LLM generates great markdown text, we're unable to display it properly on the frontend. To resolve this, we'll use the marked library to parse markdown into HTML. First, include it in your HTML header:

<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>

Next, update your updateMessage function to integrate the marked library. This will convert the streamed content from markdown to HTML in real time:

function updateMessageInChat(element, message) {
    // Convert markdown to HTML using marked.js
    const htmlContent = marked.parse(message);
    element.innerHTML = htmlContent; // Update the message content with HTML
    const chat = document.getElementById("chat-area");
    chat.scrollTop = chat.scrollHeight; // Keep the chat scrolled to the bottom
}

Now, run the code and send this message: show me the menu please. Watch how the LLM streams the response in real time and how we beautifully render the markdown.

Using LLaMA from Together AI

In one of our previous posts, we highlighted the importance of choosing the right LLM for the right job, as it directly impacts both pricing and performance. For instance, there's no need to use the latest and most expensive LLM for simple tasks like generating emails, etc.

In this section, we'll switch from OpenAI's GPT-4o Mini to Meta's LLaMA 3 8B from Together AI. Here’s a comparison of their pricing:

• GPT-4o Mini: $0.15 per million input tokens, $0.60 per million output tokens

• Meta's LLaMA 3 8B: $0.055 per million input tokens, $0.055 per million output tokens

Together AI offers a more affordable option, and it’s compatible with the OpenAI SDK for streaming content. Here’s how to make the switch:

1. Get your API keys from Together AI.

2. Update your API configuration:

    const openai = new OpenAI({
      apiKey: process.env.TOGETHER_API_KEY,
      baseURL: "https://api.together.xyz/v1",
    });
   

3. Update the model in your request

    const response = await openai.chat.completions.create({
      model: "meta-llama/Llama-3-8b-chat-hf",
      messages: [...context, ...conversations],
      stream: true,
    });
   

That’s it! You’ve successfully changed the API key, updated the URL, and switched the model. Test your app to see the new setup in action.

Conclusion

In this post, we built upon our previous work by implementing streaming for our chatbot, adding markdown parsing for enhanced content display, and exploring how to switch LLMs to optimize for cost and performance. In the next post, we'll dive into function calling with the OpenAI SDK and explore how to implement it effectively. Until then, keep coding!

In our ongoing series, we've explored the OpenAI SDK and dived into various LLMs (Large Language Models) and their SDKs. In the previous post we've covered prompting techniques gaining a solid foundation in generative AI. Now, it's time to put all that learning into practice.

In this post, we’ll build a simple yet effective pizza chatbot using the skills we've acquired. From utilizing LLM SDKs like OpenAI to applying advanced prompting techniques, we’ll bring all these elements together to create something tangible. While this chatbot won't have database connectivity—something we can add later—it serves as a great starting point to showcase the power of AI.

The inspiration for this chatbot comes from a course I previously mentioned, offered by DeepLearning.AI. The course demonstrated these concepts in a Jupyter Notebook, and I highly recommend it for anyone interested in prompting techniques. In this post, we’ll take that foundational knowledge and transform it into a fully-fledged chatbot.

Setting Up the Server

In this section, we'll set up a simple Node.js application using Express, with one endpoint for basic text generation with OpenAI. Open your terminal and run the following commands to initialize your Node.js app and install the necessary dependencies:

mkdir pizza-chatbot
cd pizza-chatbot
npm init -y
npm install express openai dotenv

Next, create a file named server.js and set up a basic Express server. Here's the code:

require("dotenv/config");
const express = require("express");
const { OpenAI } = require("openai");
const path = require("path");

const app = express();
const port = 3000;

const openai = new OpenAI({
  apiKey: process.env.OPENAI_PROJECT_KEY,
});

app.use(express.static(path.join(__dirname, "public")));
app.use(express.json());

app.get("/", (req, res) => {
  return res.sendFile(path.join(__dirname, "public", "index.html"));
});

app.listen(port, () => {
  console.log(`Server is running on http://localhost:${port}`);
});

• app.use(express.static(...)) serves static files like HTML, CSS, and JavaScript from the public directory.

• app.use(express.json()) allows us to parse incoming JSON requests.

• We create an OpenAI instance using the API key stored in our .env file.

• The app.get("/") route serves our index.html file when a user visits the root URL (/) of our application.

• The server listens on port 3000, and when it's running, you can access the app at http://localhost:3000.

Implementing the Chat Endpoint

Let's create the chat endpoint, which will handle conversations between our pizza chatbot and the user.

app.post("/api/chat", async (req, res) => {
  const conversations = req.body.conversations;

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [...context, ...conversations],
  });

  const assistantMessage = response.choices[0].message.content;

  return res.json({ message: assistantMessage });
});

• The endpoint receives an array of previous conversations from the frontend via the conversations variable. This array holds the back-and-forth messages between the user and the chatbot.

• We use OpenAI's chat.completions.create method to generate a response. The method takes in a list of messages, which includes the predefined context (the chatbot's system instructions) and the ongoing conversations.

Managing Conversations with Context

LLMs (Large Language Models) like GPT-4 are stateless, meaning they don’t remember previous interactions. To keep the conversation flowing, we need to manage the context ourselves by sending the history of the conversation with each new request. There are many advanced techniques to manage context, such as:

• Using Vector Databases: To store and retrieve relevant conversation history efficiently.

• Summarizing Previous Conversations: To avoid exceeding token limits by summarizing older parts of the conversation.

• Reducing Context Length: By trimming unnecessary details from previous interactions.

These techniques can help in more complex scenarios, but for now, we’re keeping it simple.

Crafting the System Prompt

The system prompt is a crucial part of our chatbot’s functionality. Here’s the prompt we’re using:

const context = [
  {
    role: "system",
    content: `
      You are OrderBot, an automated service designed to take orders for a pizza restaurant. Start by greeting the customer warmly. Then, proceed to collect their order, ensuring you ask for all the details like toppings, sizes, and extras to accurately identify each item on the menu.
      Once the order is complete, ask whether the customer prefers pickup or delivery. If they choose delivery, be sure to collect the delivery address. Summarize the entire order afterward and confirm if the customer wants to add anything else.
      Finally, collect the payment details. Throughout the conversation, keep your responses short, friendly, and conversational to ensure a smooth ordering experience.
      The menu includes:
      Pepperoni Pizza: $12.95 (Large), $10.00 (Medium), $7.00 (Small)
      Cheese Pizza: $10.95 (Large), $9.25 (Medium), $6.50 (Small)
      Eggplant Pizza: $11.95 (Large), $9.75 (Medium), $6.75 (Small)
      Fries: $4.50 (Large), $3.50 (Small)
      Greek Salad: $7.25
      Toppings:
      Extra Cheese: $2.00
      Mushrooms: $1.50
      Sausage: $3.00
      Canadian Bacon: $3.50
      AI Sauce: $1.50
      Peppers: $1.00
      Drinks:
      Coke: $3.00 (Large), $2.00 (Medium), $1.00 (Small)
      Sprite: $3.00 (Large), $2.00 (Medium), $1.00 (Small)
      Bottled Water: $5.00
    `,
  },
];

Breaking Down the Prompt:

• Role as a System: The prompt begins by establishing the chatbot’s role as "OrderBot," which sets the expectations for how it should behave and respond.

• Guided Interaction: It instructs the chatbot to greet the customer warmly, take their order, and ask specific questions about the order details, such as toppings and sizes.

• Customer Experience: The prompt emphasizes a friendly and conversational tone, ensuring that the chatbot’s responses are not only accurate but also pleasant for the user.

• Menu Details: The entire menu is included in the prompt, so the chatbot has all the necessary information to assist the customer in real-time.

• Handling the Order Process: The prompt outlines the steps to collect the order, summarize it, ask for pickup or delivery options, and finally, request payment details.

Prompting Technique:

• Role-based Prompting: This technique clearly defines the chatbot’s role and tasks, guiding it to perform a specific function—taking pizza orders in this case.

• Contextual Prompting: By including the entire menu and detailed instructions, the prompt provides a rich context for the chatbot, enabling it to generate more accurate and relevant responses.

This well-structured prompt ensures that the chatbot delivers a smooth and coherent user experience.

Designing the Frontend

Now that we’ve set up the backend for our chat application, let’s move on to creating the frontend. We’ll be building a simple chat interface using HTML, CSS, and JavaScript. First, create a public folder in your project directory. Inside this folder, create two separate files: index.html and styles.css. In the styles.css file, add the following CSS code:

body {
  font-family: Arial, sans-serif;
  margin: 0;
  padding: 0;
  display: flex;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  height: 100vh;
  background-color: #f0f0f0;
}

#chat {
  width: 80%;
  height: 90%;
  background: #ffffff;
  position: relative;
  padding: 1.25rem;
  border-radius: 0.5rem;
  box-shadow: 0 0 0.625rem rgba(0, 0, 0, 0.1);
  display: flex;
  flex-direction: column;
}

#chat-area {
  flex: 1;
  overflow-y: auto;
  padding: 2rem;
  padding-bottom: 3.5rem;
}

.chat-message {
  margin: 0.625rem 0;
}

.user {
  text-align: right;
}

.assistant {
  text-align: left;
  background-color: #f6f6f6;
  padding: 0.625rem;
  border-radius: 0.3125rem;
}

#message-input {
  flex: 1;
  padding: 0.625rem;
  border: 0.0625rem solid #ccc;
  border-radius: 0.5rem;
}

#send-button {
  padding: 0.625rem 1.25rem;
  border: none;
  background-color: #007bff;
  color: #ffffff;
  border-radius: 0.5rem;
  cursor: pointer;
}

#chat-form {
  position: absolute;
  bottom: 1.25rem;
  left: 1.25rem;
  right: 1.25rem;
  display: flex;
  gap: 1rem;
  padding-right: 2rem;
  padding-left: 2rem;
  align-items: center;
  background: #ffffff;
}

In the index.html file, paste the following code:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Chat</title>
    <link rel="stylesheet" href="styles.css" />
  </head>
  <body>
    <div id="chat">
      <div id="chat-area">

      </div>
      <form id="chat-form">
        <input
          type="text"
          id="message-input"
          placeholder="Enter your message here..."
          required
        />
        <button type="submit" id="send-button">Send</button>
      </form>
    </div>
  </body>
</html>

Run node server.js from your terminal and open your browser and go to http://localhost:3000 to view the chat interface.

Handling Chat Actions

Now that we have our chat interface set up, let's make it functional by sending the user's messages to the server and displaying the response from our chatbot. We’ll achieve this using JavaScript to handle the form submission and update the chat UI.

To start, we’ll add a <script> tag at the end of our index.html file and include the following JavaScript code:

<script>
  document.addEventListener("DOMContentLoaded", () => {
    const conversations = [];

    // Send an initial message when the page loads
    const initialMessage =
      "Hi there! 😊 Ready to place your pizza order 🍕 or have any questions? Let me know!";
    addMessageToChat("Assistant", initialMessage, "assistant");

    // Set up the form submit event listener
    document
      .getElementById("chat-form")
      .addEventListener("submit", async (event) => {
        event.preventDefault(); // Prevent the default form submission

        const userMessage = document.getElementById("message-input").value;
        if (!userMessage) return; // Do nothing if the input is empty

        // Add user message to chat
        addMessageToChat("User", userMessage, "user");

        // Add the user's message to the conversation
        conversations.push({ role: "user", content: userMessage });

        // Send the user message to the server
        const response = await fetch("/api/chat", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ conversations }),
        });

        const data = await response.json();
        const assistantMessage = data.message;

        // Add assistant's response to chat
        addMessageToChat("Assistant", assistantMessage, "assistant");

        // Update the conversation with the assistant's message
        conversations.push({ role: "assistant", content: assistantMessage });

        // Clear the input field
        document.getElementById("message-input").value = "";
      });
  });

  // Function to add messages to the chat
  function addMessageToChat(sender, message, className) {
    const chat = document.getElementById("chat-area");
    const messageElement = document.createElement("div");
    messageElement.className = `chat-message ${className}`;
    messageElement.innerHTML = `<strong>${sender}:</strong> <span class="message-content">${message}</span>`;
    chat.appendChild(messageElement);
    chat.scrollTop = chat.scrollHeight;
    return messageElement.querySelector(".message-content"); // Return the span where the message content is displayed
  }
</script>

• We are adding a dummy initial message from the assistant (chatbot) when the page loads to greet the user.

• We are using the fetch API to send the conversation to the server. Once the server responds, we display the assistant's message in the chat and update the conversation array.

• The addMessageToChat function takes care of adding both user and assistant messages to the chat UI. It dynamically creates a new div element for each message, appends it to the chat area, and scrolls to the latest message.

Testing the Chat Application

1. Ensure your server is running by executing node server.js in your terminal.

2. Open your browser and go to http://localhost:3000. You should see the initial greeting message. Type a message in the input field and click "Send" to see the chat in action.

3. Finish your chat with the bot by ordering a pizza and check if it follows your instructions correctly.

Conclusion

In this post, we built a basic pizza chatbot using Node.js, Express, and OpenAI. In the next post we will be:

1. Adding Streaming Responses

Currently, our chatbot sends a response all at once, which might not provide the best user experience. In a real-world application, streaming responses—where the chatbot sends responses incrementally—can make interactions feel more natural like the ChatGPT Interface. We'll implement streaming in an API and handle it on the frontend for a smoother chat experience.

2. Handling Markdown Responses

Our chatbot’s responses include markdown formatting, which can be less user-friendly. Instead of directly sending markdown, we could parse it into HTML on the frontend. This approach will make the responses more visually appealing and easier to read.

3. Cost and Model Considerations

As discussed in our previous posts, the choice of language model (LLM) can impact both cost and performance. While the gpt-4o-mini model is quite powerful, it might be more than necessary for a simple chatbot. We’ll explore alternatives like Mistral or other cost-effective models, which can help reduce both inference time and expenses.

Until then, keep coding 😊.

In our previous posts in this series, we explored the OpenAI SDK, in one of the posts we delved into LLM parameters like temperature, top-p, and top-k. We also covered various prompting techniques. In another post, we explored how to choose the right LLM by comparing pricing across providers and open-source models like Mistral and Llama. We also discussed why you might consider these alternatives for your specific needs.

For instance, when discussing pricing, we found that Mistral AI can be up to 10 times cheaper than OpenAI's ChatGPT, depending on your needs. You can use the Mistral 7B model from providers like Together AI or Groq if it fits your use case.

In this post, we'll dive into working with alternative AI SDKs, focusing on providers like Cohere, Anthropic, Together AI and models like Mistral, Llama, and more.

Together AI

Together AI is a powerful platform that offers over 100 open-source LLMs. From text generation models like Mistral to the latest LLaMA 3.1 from Facebook, and Google's Gemma 2, to cutting-edge image generation models like Stable Diffusion, Together AI has it all. Many of these models are fine-tuned versions of their originals, making them even better for specific tasks.

One example of a fine-tuned model available on Together AI is the Llama-2-7B-32K-Instruct. This model is built on Meta’s Llama-2 architecture and optimized for handling instruction-based tasks more effectively.

Getting started with Together AI is straightforward. Simply navigate to Together AI, sign in, and you'll receive $50 in free credits. Fill out some basic information, get your API keys, and save them for later use. In the playground, you can choose from a variety of models, adjust parameters like temperature, top-p, and top-k, and experiment with your prompts. You can also visit the models page to compare inference costs and make an informed decision. Below is a simple example of using the Together JS SDK.

require("dotenv/config");
const Together = require("together-ai")

const togetherAI = new Together({
  apiKey: process.env.TOGETHER_API_KEY,
});

async function main() {
  const response = await togetherAI.chat.completions.create({
    model: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
    messages: [
      {
        role: "user",
        content: "What is the capital of India?"
      }
    ],
  });

  console.log(response.choices[0].message);
}

main();

If you've been following this series, you'll notice that the code is very similar to the OpenAI SDK. This means you can easily use both OpenAI's models and Together AI in your application without a steep learning curve. You can even migrate from OpenAI to the latest open-source models with ease.

Anthrophic's Claude LLMs

Anthropic, founded in 2021 by former OpenAI members, focuses on creating safe and reliable AI systems. Backed by major companies like Amazon and Google, Anthropic has advanced its research significantly.

Their latest model, Claude 3, comes in three versions: Haiku, Sonnet, and Opus, each offering different levels of capability. With a context window of up to 200K tokens, Claude 3 can handle long conversations and complex tasks effectively. In June 2024, Anthropic released Claude 3.5 Sonnet, which is faster, more accurate, and excels at complex tasks like customer support and multi-step workflows.

Go to Anthropic's console and create your first API key. You'll get $5 in credit. Remember, Claude is a proprietary model, not open source. You can check its pricing here. Below is the code to make a request to Anthropic's LLM.

require("dotenv/config");
const Anthropic = require("@anthropic-ai/sdk");

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPHIC_API_KEY,
});

async function main() {
  const response = await anthropic.messages.create({
    model: "claude-3-5-sonnet-20240620",
    max_tokens: 1000,
    temperature: 0,
    system: "You are a teacher.",
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "What is the capital of India?",
          },
        ],
      },
    ],
  });

  console.log(response.content[0].text);
}

main();

Cohere

Cohere is a leading AI platform focused on natural language processing (NLP) for enterprise applications. In 2024, they launched Cohere Command, a suite of generative AI models designed for complex business tasks like customer support automation and content creation.

Cohere's SDK is packed with tools and connectors that enhance their language models, making it easy for businesses to integrate advanced NLP features. You can perform tasks like text generation, translation, and classification, all accessible through the Cohere dashboard.

One unique feature is "connectors," which allow you to add tools like web search, enabling the model to access the latest information from the web. Cohere is a powerful platform that I use in my applications, and I plan to create a separate tutorial to cover all its features in detail.

require("dotenv/config");
const { CohereClient } = require('cohere-ai');

const cohere = new CohereClient({
  token: process.env.COHERE_API_KEY,
});

async function main() {
  const response = await cohere.chat({
    chatHistory: [],
    message: 'What is the price of Nvidia stock?',
    // perform web search.
    connectors: [{ id: 'web-search' }],
  });

  console.log(response);
}

main();

Check the model's response, and you'll see a documents key—this contains all the information it retrieved from a web search. You can even create custom connectors to access your own data, like from OneDrive. This makes Cohere very powerful.

Mistral

Mistral AI is a French startup, founded in April 2023 by ex-Meta and Google DeepMind employees, focusing on open-source large language models (LLMs). Their model family includes the powerful Mistral 7B with 7 billion parameters, the efficient Mixtral 8x7B with up to 45 billion parameters, and the versatile Mistral Large. People love Mistral AI for its commitment to open-source, customization options via fine tuning the models, compute efficiency, and accessibility through various platforms, making advanced AI tools available to everyone.

Head to the Mistral console, sign up for a free trial, create your API keys, and follow the code below.

require("dotenv/config");
const { Mistral } = require("@mistralai/mistralai");

const mistral = new Mistral({ apiKey: process.env.MISTRAL_API_KEY });

async function main() {
  const chatResponse = await mistral.chat.complete({
    model: "mistral-large-latest",
    messages: [{ role: "user", content: "What is the capital of India?" }],
  });

  console.log("Chat:", chatResponse.choices[0].message.content);
}

main();

Conclusion

Besides the providers mentioned, we also have Anyscale AI, Eden AI, and cloud providers like AWS, Azure, and Google which offer proprietary models like ChatGPT and Anthropic’s models, along with open-source options like Mistral and Llama. No matter which model you choose, always check its capabilities, pricing, and response time. In the next tutorial, we’ll use everything we’ve learned so far to build a pizza chatbot with Node.js. 🍕🤖. Until next time, keep coding!

The quality of the output from an LLM heavily depends on the prompt you provide. The more detailed and clear your prompt is, the better the response you’ll get from the LLM. That’s why understanding how to craft good prompts is so important when working with these models. In this post, we'll dive deeper into working with the OpenAI SDK and explore some popular techniques for crafting effective prompts.

If you want to master the art of prompting, DeepLearning.AI's "ChatGPT Prompt Engineering for Beginners" is the perfect launchpad. This course breaks down the basics of prompt engineering into easy-to-digest lessons, making it accessible even if you're new to AI. You'll explore various prompting techniques that can significantly enhance the effectiveness of your interactions with ChatGPT. It's a must for anyone looking to unlock the full potential of AI in their projects.

Basics of Prompting

Let’s start with some basic prompts. Imagine we want to write a LinkedIn post about a potential business deal. Below are two prompts—one basic and one with more context —showing how different prompts can lead to different responses. You’ll see how adding context and a system message helps generate more relevant and effective results.

Basic Prompt -

const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
        role: "user",
        content: "Write a LinkedIn connection request..",
      }],
  });

Advanced Prompt -

const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
        role: 'system',
        content: 'You are a professional Linkedin user.'
      },
      {
        role: "user",
        content: "Write a personalized LinkedIn connection request to a potential business partner. Mention a shared interest in technology and set a collaborative tone.",
      }],
  });

In this advanced example, the system message “You are a professional LinkedIn user” helps the LLM generate a more tailored and context-aware connection request. By providing specific details in the prompt, we guide the LLM to produce a message that is not only more engaging but also aligns with the intended purpose. Give the code a try to see the generated response. You can also experiment with the temperature setting to get different results. The OpenAI Playground provides a user-friendly interface for this.

How to Write Good Prompts:

1. Your Prompt should comprise of the main goal: What you want the LLM to do and Extra info: Details to help the LLM understand better.

2. Give the AI a role (if needed): Example: You're a Twitter expert who writes popular tweets. Write a tweet about hiking.

3. Add extra rules or information: Example: Write a tweet about hiking. Use no more than two emojis. Focus on nature lovers. Mention 2 benefits of hiking often.

4. LLM Output is a starting point, fine tune and adjust it as needed. Tell it what parts need to be improved.

Add Meaningful Context to your Prompts:

• Prefer short, focused sentences.

• Add important keywords & avoid unnecessary information.

• Define the target audience (for tweets, LinkedIn posts, blog posts)

• Control tone, style & length of the output.

• You can also control the output format (JSON, etc.)

Zero Shot Prompting

Zero-shot prompting is a way to use LLM without giving it special examples. You simply ask the LLM to do a task, and it tries its best using what it already knows. Use zero-shot prompting when:

• You need a quick answer.

• The task is simple.

const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
        role: "user",
        content: "Tell me how AI helps in hospitals.",
      }],
  });

In the above example, the LLM gives a helpful answer just from your question, without needing extra information or examples.

One-Shot Prompting

One-shot prompting is when you give the LLM one example to help it understand what you want. It's like showing someone how to do something once before asking them to try it themselves. Use one-shot prompting when:

• You want the LLM to follow a specific style.

• The task is a bit tricky.

• You need to guide the LLM without giving too many instructions.

const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
        role: 'system',
        content: `You are a professional LinkedIn user. 
        Your task is to write a LinkedIn post announcing new events like the below example.
        Example: 'Greetings from XYZ Corp. Excited to announce our new partnership with XYZ Corp. Together, we're driving innovation in the tech industry!`
      },
      {
        role: "user",
        content: "Now write a LinkedIn post announcing a new product launch.",
      }],
  });

By giving one example, you help the LLM understand the tone and style you want, so it can give you a better answer.

Few-Shot Prompting

Few-shot prompting is when you give the LLM a handful of examples before asking it to do a task. It's like showing someone how to do something a few times before they try it on their own. Use few-shot prompting when:

• You want the LLM to follow a specific pattern

• You need the LLM to sort things into categories

• You're working on tasks where examples really help, like translating languages in a particular style or particular tone, e.g. Translate some German text into Shakespearian English.

const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
        role: 'system',
        content: `You are a professional news reader. 
        Your task is to categorize news articles the below examples.
        Article 1: "New electric car saves energy and drives far." Topic: Technology
        Artilce 2: "Latest clothing styles are changing how people dress on the street." Topic: Fashion`
      },
      {
        role: "user",
        content: `What's the topic of this article: "Meditation helps people feel less stressed and focus better".`,
      }],
  });

By showing a few examples, you help the LLM understand how to sort articles into the right topics.

Chain-of-Thought Prompting

Chain-of-Thought prompting is like asking the AI to "show its work." Instead of just giving an answer, the AI explains how it got there, step by step. Use Chain-of-Thought prompting when:

• You have a tricky problem to solve.

• You want to see how the AI thinks and want to verify it.

• The task has multiple steps.

const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
        role: "user",
        content: `Solve the following math problem and show your work: What is 25% of 80?`,
      }],
  });

By explaining each step, the AI shows you its thinking process. This helps you understand the answer better and makes sure the AI is on the right track.

Prompts with Output Templates

Output templates are like forms you give the AI to fill out. They help you get information in a neat, organized way. Use output templates when:

• You want information in the same format every time

• You need to compare different things easily

• You're making lists or collecting specific details

const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
        role: "user",
        content: `Tell me about 5 great summer vacation spots. Use this form:
            Place:
            When to go:
            How warm it gets:
            How much sun you'll see:
            How often it rains:
        `,
      }],
  });

By using a template, you get all the information you want in a tidy, easy-to-read format.

Perspective Prompting

Perspective prompting involves framing your prompt from a specific point of view or role. By doing this, you guide the LLM to generate responses that align with a particular perspective, making the output more tailored and relevant. Use perspective prompting when:

• You want answers from a specific point of view

• You need to understand different roles or experiences

• You're creating content for particular groups of people

const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
        role: "user",
        content: `Create a travel plan for a 2-week yoga retreat in Austria from the perspective of a yoga trainer.`,
      }],
  });

Laddering Prompting

Laddering prompting involves breaking down a complex task into smaller, manageable prompts. Instead of tackling everything at once, you handle each part step by step. This approach helps in building detailed and accurate responses, especially for intricate projects. Use laddering prompting when:

1. The task is too big or complex for one question

2. You need to break down a problem into smaller steps

3. You want a well-organized, step-by-step answer

For example, if you're writing a long report, you might first ask about the main topics, then about each topic in detail, and finally how to put it all together.

Another example, imagine you want to create a complete REST API. Instead of trying to do it all in one prompt, you can break it down into smaller steps, setup the project, connect to the database, create schemas, create endpoints, etc.

Delimiters in Prompts

Delimiters are used to clearly separate different parts of a prompt, making it easier for AI to understand and respond accurately. Common delimiters include triple quotation marks (""") and XML tags (<tag>). These helps structure the prompt, ensuring that instructions, examples, and other information are distinct and easy to follow.

Example 1: Triple Quotation Marks

Triple quotation marks are useful for enclosing larger blocks of text. They help in clearly demarcating sections within a prompt.

Instruction: Please summarize the text delimited by """.
""" your text .... """

Example 2: XML Tags

XML tags provide a more structured way to label different parts of a prompt. This method is particularly useful for complex prompts with multiple sections.

<instruction>
Please summarize the following text.
</instruction>
<text>
your text ....
</text>

Enhancing your Prompts

If you're ever unsure about crafting the perfect prompt, why not let the LLM help you out? You can ask the model itself to generate prompts, making the process much easier. Once you've got your prompts, it's crucial to compare the outputs to see which ones deliver the best results. Tools like Anthropic's workbench are great for this—allowing you to generate, test, and compare prompts in one powerful platform. You can even run test cases to fine-tune your prompts. If you're curious about how different LLMs respond to the same prompts, Airtrain.ai lets you compare outputs side by side, helping you choose the best LLM for your needs. And for managing and observing your prompts, Pezzo offers a comprehensive platform to keep everything organized and optimized.

Conclusion

In this article, we explored various prompting techniques and discussed how to tailor prompts for better results with LLMs. To reiterate, the prompt you define directly influences the output you get from the model. Crafting effective prompts is crucial for your day-to-day tasks and applications.

Make sure to include relevant context in your prompts to guide the LLM effectively. Don't hesitate to experiment with different prompts and compare multiple responses to find the best outcome. LLMs are versatile tools used for various tasks, including translation, content creation, and more. Tailor your prompts to fit each specific use case to maximize their effectiveness.

In the next post, we'll dive into using other LLM SDKs. We'll explore how to work with open-source models such as Mistral and Llama, as well as proprietary LLMs offered by Cohere and Anthropic. Until then, keep coding!

1. Memory in LLMs: How to keep context in chatbots.

2. Temperature: What it is and how it affects LLM responses.

3. Top-P and Top-K: Tweaking these to improve LLM responses.

While we're using OpenAI's SDK, these concepts apply to most LLMs and their tools.

Memory in LLMs: Understanding Context and Pricing

LLMs Don't Have Built-in Memory. LLMs don't remember previous conversations on their own. They're designed to generate text based on the input they receive at that moment. This means each time you ask a question, it's like starting a new conversation. Let's look at an example:

async function main() {
  const responseOne = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content: "You are a friendly teacher, teaching 4 year olds.",
      },
      {
        role: "user",
        content: "What is the capital of India?",
      },
    ],
  });

  console.log("First Response", responseOne.choices[0].message);

  const responseTwo = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "user",
        content: "How large is it?",
      },
    ],
  });

  console.log("Second Response", responseTwo.choices[0].message);
}

The responses:

First Response {
  role: 'assistant',
  content: "The capital of India is New Delhi! It's a big city where the government works and many important decisions are made. Do you want to know more about New Delhi?",
  refusal: null
}
Second Response {
  role: 'assistant',
  content: 'Could you please provide more context or specify what "it" refers to? That way, I can give you a more accurate answer.',
  refusal: null
}

As you can see, the second response doesn't know we were talking about New Delhi. The LLM treats each request as brand new.

Managing Memory: The Developer's Job

async function main() {
  const messages = [
    {
      role: "system",
      content: "You are a friendly teacher, teaching 4 year olds.",
    },
  ];

  messages.push({
    role: "user",
    content: "What is the capital of India?",
  });

  const responseOne = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages,
  });

  console.log("First Response", responseOne.choices[0].message);

  messages.push({
    role: responseOne.choices[0].message.role,
    content: responseOne.choices[0].message.content,
  });

  messages.push({
    role: "user",
    content: "How large is it ?",
  });

  const responseTwo = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages,
  });

  console.log("Second Response", responseTwo.choices[0].message);
}

Above is a simple way to maintain context in our conversations with the LLM. We can keep track of all messages in a messages array, which includes both user inputs and AI responses. Whenever we make a new request to the LLM, we send the entire messages array along with it.

This method is quite straightforward. It’s easy to implement, and it ensures that the LLM has access to the full conversation history. But there is an issue. As conversations get longer, passing the entire history to the LLM for each new message increases the token count. This directly impacts your costs. For example, after 20 messages, you'd be sending all 20 previous messages plus the new one each time.

There are more sophisticated ways to handle memory, like using libraries such as LangChain's Memory feature. These can help manage long conversations more efficiently, but they're complex enough to warrant their own tutorial in the future.

Other LLM providers, like Cohere, make it even easier. Their SDK allows you to simply add a conversation_id, and it will automatically manage the conversation history and context for you.

Few approaches to LLM memory management as highlighted below -

Pricing Control: Limiting Tokens

async function main() {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content: "You are a friendly teacher, teaching 4 year olds.",
      },
      {
        role: "user",
        content: "What is the capital of India?",
      },
    ],
  });

  console.dir(response, { depth: null, colors: true });
}

I have discussed in depth about tokens, in my previous post. Tokens are the basic units that LLMs process. They can be parts of words, whole words, or even punctuation marks. LLMs charge based on the number of tokens used, both in your input (prompt) and in the generated output. In the example above we are printing the whole response object -

{
  id: 'chatcmpl-9wUBsQCvYQGWOvZif7UWrPl0vPefs',
  object: 'chat.completion',
  created: 1723726104,
  model: 'gpt-4o-mini-2024-07-18',
  choices: [
    {
      index: 0,
      message: {
        role: 'assistant',
        content: "The capital of India is New Delhi! It's a big city with lots of important buildings and places. Would you like to know more about India or its culture?",
        refusal: null
      },
      logprobs: null,
      finish_reason: 'stop'
    }
  ],
  usage: { prompt_tokens: 30, completion_tokens: 32, total_tokens: 62 },
  system_fingerprint: 'fp_507c9469a1'
}

In the LLM's response, you can see the usage. For example, if it shows 62 tokens were used, that's what you'll be charged for. You can check the pricing for Open AI models here.

To check token counts before sending requests:

1. Use OpenAI's 'tiktoken' package.

2. Or use this handy tool: OpenAI Platform Token Counter.

When building apps with LLMs, it's crucial to manage costs. Here are two ways to limit token usage:

1. Limiting Input Tokens: Use the 'tiktoken' package to count tokens in your prompt. Adjust your prompt if it's too long.

2. Limiting Output Tokens: Use the 'max_tokens' parameter in your API call. This sets a cap on how many tokens the LLM can generate in its response

const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [/* your messages here */],
  max_tokens: 100  // This limits the response to 100 tokens
});

Why Limit Tokens?

• Cost Control: Prevent unexpected high bills.

• Performance: Shorter responses can be generated faster.

• Focused Answers: Encourages more concise, targeted responses.

Remember, finding the right balance is key. Too few tokens might cut off important information, while too many could lead to unnecessary costs.

Fine-tuning LLM Responses: What is LLM Temperature?

As discussed in the previous post LLMs are text generation models. It starts by reading the context you've provided (prompt), much like reading a book and trying to guess the next word.

Example: If the LLM is completing the sentence "The sky is ______," it might assign probabilities like this:

• "blue" (40% chance)

• "cloudy" (30% chance)

• "clear" (20% chance)

• "beautiful" (10% chance)

The LLM is likely to choose "blue," but sometimes it might surprise you. This variety makes LLM responses feel natural and diverse.

Temperature is a setting that controls how creative or predictable an AI's responses will be. Think of it like a creativity knob you can adjust. How Temperature Works:

1. Low Temperature (0 to 0.5): The AI plays it safe. It gives predictable, common answers. Good for tasks needing accurate, consistent responses. Example: Answering factual questions, a quiz chatbot, etc.

2. High Temperature (0.5 to 1): The AI gets more creative. It might give unusual or unexpected answers. Good for tasks needing originality. Example: Brainstorming ideas, writing stories, a sales pitch chatbot, etc.

Example in code:

const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [/* your messages here */],
  temperature: 0.7  // Adjust this value between 0 and 1
});

• Low temperature doesn't mean "better" - it depends on what you need.

• Experiment with different settings to find what works best for your task.

By adjusting temperature, you can fine-tune the AI's responses to better suit your needs, whether you're looking for creativity or consistency. You can try playing around with different temperature values for your prompts at the OpenAI Playgournd.

Fine-tuning LLM Responses: Top-P & Top-K parameters

Now, let’s talk about top-p and top-k, which influence how the LLM chooses words.

Top-P (Nucleus Sampling): Imagine you're at an ice cream shop, and the flavors represent words. Top-P is like saying, "I'll only look at flavors that make up 75% of all sales." This approach is flexible—you might consider just a few super popular flavors or a larger mix.

Example: If top-p is 0.75 and you have:

• Vanilla (40% popularity)

• Chocolate (35% popularity)

• Strawberry (15% popularity)

• Mint (10% popularity)

You'd only consider Vanilla, Chocolate, and Strawberry because they add up to over 75%. This is how the LLM will generate text, if you set the top_p parameter.

const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [/* your messages here */],
  top_p: 0.75
});

Top-K: On the other hand, top-k is like saying, "I'll only look at the top 3 most popular flavors," regardless of their exact popularity. Example - If top-k is 3, you always consider Vanilla, Chocolate, and Strawberry. This is how the LLM will generate text, if you set the top_K parameter. You can play with all these parameters in the Open AI Playground. The OpenAI SDK doesn't offer a top-k setting, but other LLM providers do, like Anthropic's Claude.

const response = await anthropic.messages.create({
  model: "claude-3-5-sonnet-20240620",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Hello, Claude" }],
  top_K: 50
});

How They Differ from Temperature: While top-p and top-k decide which words the LLM can choose from, the temperature setting adjusts how "daring" the AI is in making its choices.

Conclusion

In this tutorial, we explored OpenAI SDK, focusing on memory, conversational context, and tokens. We discussed how tokens impact pricing and how to fine-tune LLM responses using parameters like temperature, Top-P, and Top-K, all with the popular OpenAI SDK. In the next tutorial, we'll dive into some prompting techniques with OpenAI SDK. Until then, keep coding!

Getting Your OpenAI Project Keys

First things first—let’s get you set up with the necessary credentials. Head over to the OpenAI platform and log in using your ChatGPT credentials (or create an account if you’re new). Once logged in, go to User Settings > OpenAI API. You might need to set up billing—it’s a quick process and just a formality to get started.

OpenAI organizes your work neatly: your account can house multiple projects under one organization, which is great for managing different AI applications.

Now, let’s create your Project Key. Navigate to API Keys > OpenAI API and click "Create secret key." Select your project, and voilà! Your key is ready. Make sure to copy and store it safely—you won’t be able to see it again. With this key in hand, you’re all set to start working with the OpenAI SDK.

Making a Request with the OpenAI SDK

Now for the fun part—let’s get your Node.js project set up. Follow these steps to get up and running:

1. Install the necessary packages:

    npm install openai dotenv
   

2. Create a .env file in the root of your project, and add your OpenAI Project key:

    OPENAI_PROJECT_KEY=your_openai_project_key_here
   

3. Create a file named app.js and paste the following code:

    require("dotenv/config");
    const { OpenAI } = require("openai");
   
    const openai = new OpenAI({
      apiKey: process.env.OPENAI_PROJECT_KEY,
    });
   
    async function main() {
      const response = await openai.chat.completions.create({
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: "What is the capital of India?" }],
      });
   
      console.log("AI Response", response.choices[0].message);
    }
   
    main();
   

4. After you run your code, you should see something like this in your terminal:

    AI Response {
      role: 'assistant',
      content: 'The capital of India is New Delhi.',
      refusal: null
    }
   

Congratulations! You’ve just made your first AI call. Let’s break down what’s happening:

• Model: When working with the OpenAI SDK, it’s important to select the right LLM for your needs. OpenAI offers various models. Each model has unique strengths, context length, and pricing. For instance, GPT-3.5 is great for text generation, while gpt-4o and gpt-4o-mini can handle both text and images. Check the pricing for each model here. Consider your project’s requirements before choosing a model. I highly recommend you read my previous post on this topic.

• Making the Request: You asked the AI a question—“What is the capital of India?”—which you sent in an array of messages with the role set to 'user'. This role tells the AI that the message is coming from a user.

• The Response: The AI replies with a message where the role is 'assistant'. This indicates the response is from the AI, and the content contains the actual answer: “The capital of India is New Delhi.”

• The Role Parameter: The role parameter is key to managing conversations. By setting role: 'user', you tell the AI that the message is from the user. When the AI responds, it uses role: 'assistant' to indicate that it’s replying. This system helps structure interactions.

Customizing Roles

You can add more depth to your interactions by customizing roles. For example, you can introduce a 'system' role to guide the AI’s behavior:

async function main() {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
        { role: "system", content: "You are a friendly teacher, teaching 4-year-olds." },
        { role: "user", content: "What is the capital of India?" }
    ],
  });

  console.log("AI Response", response.choices[0].message);
}

Here’s what the AI might say:

AI Response {
  role: 'assistant',
  content: "The capital of India is New Delhi! It's a big city where important government buildings are located. Have you ever seen pictures of New Delhi? It has beautiful parks and monuments!",
  refusal: null
}

In this example, the system role acts as a guide, setting the context for the AI. By telling the AI to be a friendly teacher, you influence how it responds to the user’s question. The system role is handy for scenarios where you need consistent behavior from the AI. For example:

• Customer Support: Set the AI to be patient and empathetic.

• Educational Tool: Instruct the AI to be detailed and thorough in its explanations.

• Creative Writing: Guide the AI to adopt a particular style or tone, like being humorous or serious.

Working with Multimodal LLMs

Using multimodal LLMs like gpt-4o-mini is straightforward. These models can handle both text and images, offering a wide range of applications. Here’s how you can work with an image in your AI request:

require("dotenv/config");
const { OpenAI } = require("openai");
const fs = require("fs");

const openai = new OpenAI({
  apiKey: process.env.OPENAI_PROJECT_KEY,
});

async function main() {
  const base64Image = Buffer.from(fs.readFileSync("laptop.jpeg")).toString("base64");

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content: "You are a friendly teacher, teaching 4-year-olds.",
      },
      {
        role: "user",
        content: [
          { type: "text", text: "What is this image about?" },
          {
            type: "image_url",
            image_url: { url: `data:image/png;base64,${base64Image}` },
          },
        ],
      },
    ],
  });

  console.log("AI Response", response.choices[0].message);
}

Here’s an example of what the AI might say:

AI Response {
  role: 'assistant',
  content: "The image shows a laptop, specifically a MacBook Pro, with a scenic nature background displayed on the screen. The background seems to depict hills and trees, suggesting a beautiful outdoor landscape. The laptop's keyboard and trackpad are visible as well. It appears to be resting on a colorful tablecloth.",
  refusal: null
}

In this case, the content for the user is an array of objects, containing both text and image data. The image_url can either be a base64 encoded string for images stored on your device or a regular URL for images from the web.

Streaming

The stream parameter is an option you can use when making requests to the OpenAI API. By default, when you request a completion from the API, the entire response is generated and sent back to you all at once. This can take some time, especially for longer responses.

When you set the stream parameter to true, the API sends the response back in smaller chunks as they are generated. This means you can start processing or displaying the response before the entire completion is finished.

You might want to use the stream parameter in situations where providing real-time feedback is crucial, such as in chat applications where streaming can make interactions feel more responsive. It’s also beneficial for tasks that generate long responses, as it allows you to start processing the data sooner rather than waiting for the entire response to be generated. Additionally, streaming can significantly enhance the user experience by reducing perceived latency, making your application feel faster and more interactive.

async function main() {
  const stream = await openAI.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
        { role: "system", content: "You are a friendly teacher, teaching 4-year-olds." },
        { role: "user", content: "What is the capital of India?" }
    ],
    stream: true,
  });

  for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta?.content || "");
  }
}

main()

Conclusion

In this post, we finally got our hands dirty with some code, starting with the OpenAI Node.js SDK. I encourage you to experiment with different models and prompts to familiarize yourself with the SDK. In our next post, we’ll dive deeper into the OpenAI SDK, exploring LLM memory, tokens, temperature, and other parameters you can tweak. Until next time, happy coding!

The world of LLMs is vast and growing rapidly. We have open source LLMs like Mistral from Mistral AI, LLaMA from Facebook. We have Proprietary LLMs like GPT-4 from OpenAI, PaLM from Google. LLMs generating images like Stable Diffusion models. Multimodal LLMs (working with text, speech, and images) like Gemini from Google, GPT-4o from OpenAI. Other LLM providers like Claude AI from Anthropic, Cohere AI. Cloud providers offering both open source and proprietary AI service like Azure AI Studio, Amazon Bedrock, Google Cloud Vertex AI. Finally we have LLM platforms like HugginFace, AnyScale, Together AI, Groq offering a wide range of LLM solutions.

As JavaScript developers new to this field, choosing the right LLM can seem overwhelming. Let's dive deeper into this topic and explore how to make the best choice for your application and use case.

Some Key Factors in Choosing an AI Model:

Modality: Decide if you need a model that works with text, images, or both. For example, if you're building a chatbot, a text-based model like GPT-4 might suffice. But if you're creating an app that generates images from text descriptions, you'd need a text-to-image model like DALL-E.

If your application requires handling text, images, and audio simultaneously, consider a multimodal model like GPT-4o. While the examples mentioned are from OpenAI, don't limit yourself to one provider. Explore offerings from other companies like Anthropic's or Cohere's offerings. Remember, choosing the right modality is crucial as it significantly impacts both pricing and performance.

Model Size: Larger models generally perform better but need more computing power. Think of it like engine size in cars - a bigger engine (more parameters) usually means more power, but also higher fuel consumption. For instance, a 1 billion parameter model might work for simple tasks, while complex reasoning might require models with 100+ billion parameters.

However, it's crucial to match the model size to your specific use case. For example, if you're building a spam filter for emails, you don't need a massive model like GPT-4 (with its 1.76 trillion parameters). A smaller model like Mistral 7B, with 7 billion parameters, could be more than sufficient for such a task.

While larger models often excel at complex tasks, they may be overkill for simpler applications. Smaller models are generally much cheaper to run. They require less computing power and memory, which translates to lower operational costs. Smaller models typically provide faster inference times, which can be crucial for real-time applications like chatbots or content moderation systems.

Task-Specific Models: For specialized tasks, look for models trained specifically for that purpose. They can often outperform larger, general-purpose models at a lower cost. For example this model from llmware, providing a small, fast inference for high-quality summarizations of complex business documents.

Accuracy vs. Speed: Continuing from the previous point, larger models often provide higher accuracy but at the cost of slower processing times. The best choice depends on your specific application's requirements.

For a real-time translation app, you might prioritize speed over perfect accuracy. Users typically need quick translations during conversations or while reading, and small inaccuracies are usually tolerable. In this case, a smaller, faster model could be more suitable.

Conversely, for a medical diagnosis assistant, accuracy is paramount, even if it takes a bit longer to process. In this scenario, using a larger, more comprehensive model would be justified, as the stakes are much higher and even small errors could have serious consequences.

Deployment Options: Your choice depends on factors like your technical capabilities, budget, data privacy requirements, and specific model needs. For example, a small startup might start with a cloud service for quick deployment, while a large healthcare company might opt for on-premises deployment to ensure data privacy.

Services like Azure AI Studio, AWS Bedrock, or Google Vertex AI offer ready-to-use infrastructure. These are convenient if you're already using these cloud providers.

If you prefer full control or have strict data privacy requirements, you might deploy on your own hardware. This involves setting up and maintaining your infrastructure, including purchasing GPUs to run models efficiently. You could use open-source models like Meta's LLaMA in this scenario. This option requires more technical expertise but offers maximum control.

Services like Novita AI provide GPU resources for training, fine-tuning, and deploying models. Companies like Together AI, Anyscale AI, Portkey, Abacus AI, Groq and Eden AI offer platforms to use various open-source models. These can be good options if you are starting out.

Evaluating Large Language Models

Evaluating LLMs can be complex, with many detailed resources available online. However, for JavaScript developers new to LLMs, the process doesn't have to be overwhelming. Here's a simplified approach:

1. Focus on your project's specific needs. For example, if you're building a chatbot, test different models with sample conversations.

2. Ask yourself these key questions:

   • Does the model understand the context?

   • Are its responses helpful and relevant?

   • Is it fast enough for your application?

3. Leverage existing APIs from providers like OpenAI or Hugging Face. These often come with JavaScript SDKs, making it easier to experiment without dealing with underlying complexities.

4. Use comparison tools to evaluate different models. For instance, Airtrain AI's LLM playground is a free tool that allows you to compare model outputs side by side. You can also checkout Artificial Analysis for more detailed comparisons. Remember, the best model for your project is the one that works well for your specific use case.

For those interested in more in-depth evaluation, resources like the Hugging Face Open LLM Leaderboard or articles from Confident AI and SingleStore provide comprehensive guides on LLM evaluation metrics and methods. However, for beginners, starting with practical testing and gradually expanding your evaluation criteria as you gain experience is often the most effective approach.

Practical Tips for Implementation

Starting simple and gradually increasing complexity as you learn is often the best approach for beginners. This strategy allows you to gain practical experience while managing costs and complexity effectively:

1. Find Working Examples: Search GitHub and online tutorials for existing implementations. This can give you a head start and show you best practices.

2. Use Labeled Data: Create small, relevant datasets (30-100 samples) to test model performance for your specific use case. This helps you evaluate real-world effectiveness.

3. Start Small: Begin with smaller models and scale up as needed. This approach is similar to optimizing JavaScript applications and can significantly reduce costs.

4. Plan for Updates: Design your system to easily switch between different models. For example, if you start with GPT-4 but later want to try Claude from Anthropic, make this transition smooth. Platforms like Portkey offer unified SDKs that allow you to use multiple AI providers with the same codebase, making it easier to experiment and optimize.

Understanding Context Length in LLMs

LLMs are stateless, meaning they don’t retain memory of previous conversations. Each interaction is independent, making the concept of a context window crucial.

A context window is the maximum amount of text (measured in tokens or words) that an LLM can process in a single interaction. The size of the context window determines how much information the model can consider when generating responses.

Comparing Popular LLMs:

• Long Conversations: If the conversation exceeds the context window, older parts may be cut off. Chat clients solve this by selectively including relevant history in each prompt.

• Large Text Summarization: Texts longer than the context window are "chunked"—summarized in parts and then combined.

Pricing

Let's address the crucial aspect of pricing. As mentioned in my previous post, LLM usage is priced based on tokens. Pricing often (not always) differs for input tokens (your prompts) and output tokens (the model's responses). You're charged for both input and output tokens.

For example, comparing GPT-4o (from OpenAI) with Mistral 7B (offered by Together AI):

• GPT-4o: Input: $5.00 per 1M tokens & Output: $15.00 per 1M tokens

• Mistral 7B: $0.20 per 1M tokens (both input and output)

• Mistral 7B is 25 times cheaper for input tokens, 75 times cheaper for output tokens. On average, 50 times cheaper when considering both input and output tokens equally.

• This significant price difference highlights why choosing the right model for your use case is crucial.

Key Pricing Considerations:

1. Model Size vs. Cost: Larger models generally cost more. Ensure you're not using an overpowered (and overpriced) model for simple tasks. For instance, using GPT-4o for basic text classification would be unnecessarily expensive. Begin with smaller, more cost-effective models that can handle your task adequately. If you need GPT-4o's capabilities, consider starting with its more economical smaller version GPT-4o mini (launched by OpenAI recently).

2. Compare Providers: Even for the same open-source model (like Mistral 7B or LLaMA), prices vary across platforms. Compare offerings from providers like Together AI, Anakin, etc. Consider both pricing and performance (latency, uptime).

3. Cloud Provider Offerings: If using Azure AI Studio, AWS Bedrock, or Google Vertex AI, compare their LLM offerings. These platforms often provide a range of models at different price points.

4. Optimizing Token Usage: Be mindful of how you structure prompts. Efficient prompts can reduce token count, lowering costs.

5. Volume Discounts: For high-volume usage, many providers may offer discounted rates. Factor this in for long-term projects.

6. Pricing Calculators: Use pricing calculators like YourGPT or LLMPricing to estimate costs across different models and providers.

Remember, the cheapest option isn't always the best. Balance cost with performance, reliability, and your specific needs. Start with smaller, cost-effective models and scale up as necessary, continuously monitoring your usage and costs.

Conclusion

We've covered a lot of ground in this post, from understanding different types of LLMs to evaluating their performance and considering crucial factors like pricing. Choosing the right LLM for your project involves balancing capabilities, costs, and practical implementation details.

Remember, the best model isn't always the largest or most expensive, but the one that best fits your specific use case and budget. Start small, experiment, and scale as needed.

In our next post, we'll get our hands dirty with some actual code, exploring how to use the OpenAI JavaScript SDK to bring the power of LLMs into your projects. Get ready to turn all this knowledge into action!

In the first tutorial of our series, we explored foundation models and Large Language Models (LLMs). We took a glimpse at their basics, evolution, and use cases. In this part, we'll dive deeper into how they work. We'll explore how a computer program can be intelligent enough to produce emails, translate text, summarize data, and understand our language - all without the typical machine learning complexities.

LLMs are not human; they simply predict and generate text based on patterns. As a fellow JavaScript developer, I'll explain how this "magic" is possible in terms we can easily understand. We'll uncover how these seemingly intelligent programs process and generate human-like text.

How do LLMs Work?

Large Language Models (LLMs) are advanced AI systems designed to understand and generate human-like text. They learn from vast amounts of text data using a method called self-supervised learning. Here’s a simple breakdown of how they work:

Key Steps in LLM Functioning:

1. Input Processing:

   • When you type something, the LLM first breaks down your text into smaller pieces called tokens. For example, the sentence "Hello world!" might be split into tokens like ["Hello", "world", "!"].

2. Vector Conversion:

   • These tokens are then converted into vectors. Think of vectors as lists of numbers that represent the tokens in a way the model can understand. For instance, the token "Hello" might be represented as a vector like [0.1, 0.2, 0.3, 0.4].

3. Model Processing:

   • The LLM processes these vectors using its neural network, which is often based on a transformer architecture. This step is where the model uses its knowledge to understand the context and relationships between the tokens.

4. Output Generation:

   • The model generates a response in the form of new vectors. These output vectors are the model's predictions of what the next tokens should be.

5. Text Conversion:

   • Finally, the output vectors are converted back into tokens, and these tokens are combined to form human-readable text. So, the vectors might be translated back into words or parts of words to create a sentence.

Throughout this process, LLMs use their training on massive datasets to predict the most likely next word or sequence of words. This allows them to generate responses that are coherent and make sense in the given context.

Understanding Tokens in LLMs

Tokens are the fundamental units that Large Language Models (LLMs) use to process language. Tokenization is the process of breaking down text into these smaller, manageable units. Tokens can be words, subwords, or even characters, serving as the basic elements for processing language in LLMs.

When a user inputs a prompt, the LLM's tokenizer converts the text into a series of tokens. These tokens are typically represented as numbers, with each token corresponding to a unique identifier in the model's vocabulary. This numerical representation allows the LLM to process and manipulate text efficiently. Every LLM has a different tokenizer. For example, you can experiment with the tokenizers used by GPT-3.5 and GPT-4 here.

Understanding tokens is important, especially if you are integrating LLMs like ChatGPT into your application. This is because you are charged based on both the input and output tokens – the text you input and the text generated by the model. We will dive deeper into billing in the next post, but this is how tokens play a crucial role in it.

Vectors and Embeddings in Large Language Models (LLMs)

Vectors are lists of numbers used to represent words or tokens in LLMs. Think of them as numerical versions of words. For example, "cat" might be represented as [0.2, 0.4, 0.6]. Embeddings are a specific type of vector used in LLMs. They are dense, continuous representations of words or tokens that capture meaning.

Key Features:

1. High-Dimensional Space: Vectors and embeddings exist in a space with many dimensions, allowing LLMs to capture complex relationships between words.

2. Meaning Capture: Words with similar meanings have similar vectors. For example, "king" and "queen" might have close vector representations.

3. Mathematical Operations: Machine learning models can now perform operations on these vectors to understand language. For example: "king" - "man" + "woman" ≈ "queen"

Similarity Searches in LLMs

Imagine you're trying to find a book in a massive library where every book is represented by a unique code. That's similar to how LLMs use embeddings to find relevant information. The actual process is very complex, involving many calculations and adjustments, but here is the basic idea:

1. Convert Text to Vector: The LLM turns your search phrase into a special code (embedding). Example: You search for "healthy breakfast ideas" → [0.2, 0.5, 0.8]

2. Compare Embeddings: It compares your search codes. Example: It might find these codes:

   • "Nutritious morning meals" → [0.3, 0.4, 0.7]

   • "Fast food options" → [0.1, 0.2, 0.3]

3. Measure Closeness: The LLM calculates how similar these codes are to your search. Example: "Nutritious morning meals" is closer to "healthy breakfast ideas" than "Fast food options".

4. Retrieve Information: It presents the most closely related information.

This process helps LLMs quickly find relevant information from vast amounts of data. This is the main reason why LLMs use vectors—machines understand numbers and don’t have consciousness like humans. By using vectors, LLMs can efficiently match and retrieve information. This is how ChatGPT answers your questions so effectively. Pretty neat, right?

Conclusion

In conclusion, understanding the intricate workings of LLMs, from tokenization to embeddings, provides valuable insights into how these powerful AI systems process and generate human-like text. In the next tutorial, we will cover how to choose the right LLM for your use case. We’ll explore the different LLMs available in the market, their use cases, pricing, and more. See you in the next one!

Hey there, fellow JS developers!

Artificial Intelligence, particularly Generative AI (Gen AI), is revolutionizing our industry. We're already using tools like ChatGPT for code generation and image creation, but AI's potential goes far beyond these applications.

As JavaScript developers, it's crucial to understand how we can harness the power of Gen AI in our own applications. How can we integrate ChatGPT-like functionality into our projects? How can we leverage AI to enhance user experiences and solve complex problems?

In this series, we'll dive deep into the world of Gen AI from a JS developer's perspective. We'll cover the basics, explore how it works, and learn to implement Gen AI features in our applications. Whether you're an AI novice or looking to expand your skills, this series aims to equip you with practical knowledge and insights. Let's kick things off by introducing Gen AI and its fundamental concepts.

What is a Foundation Model?

Foundation Models represent a paradigm shift in machine learning. Unlike traditional models designed for specific tasks, these large-scale neural networks are trained on vast amounts of data, enabling them to perform a wide array of functions. The power of Foundation Models lies in their ability to process and "understand" enormous volumes of information in a fraction of the time it would take a human.

To illustrate the efficiency of Foundation Models, consider this: reading everything on the internet would take a human approximately 255,000 years, while a Foundation Model can accomplish this feat in just a few months. This incredible efficiency is what makes Foundation Models so exciting and powerful.

Foundation Models learn natural patterns from data, enabling them to perform tasks like classification, question-answering, and summarization without specific training for each task, making them highly efficient for various applications. Unlike traditional models designed for a single task, Foundation Models can handle multiple tasks simultaneously by learning general data representations. For instance, while a traditional model might only do sentiment analysis, a Foundation Model can perform sentiment analysis, language translation, and text generation all at once. This multi-task capability comes from their training on huge amount of diverse datasets, allowing them to capture complex relationships and transfer knowledge across different domains.

What are Large Language Models and their Evolution?

Large Language Models (LLMs) are a specific type of Foundation Model designed to understand and generate human-like text. The most famous example is ChatGPT, an impressive chatbot that can engage in human-like conversations on various topics.

The evolution of LLMs can be traced back to 2017 with the introduction of the Transformer architecture. This was followed by the development of BERT (Bidirectional Encoder Representations from Transformers) and GPT-1 (Generative Pre-trained Transformer) in 2018, which set new benchmarks in natural language processing tasks. Since then, we've seen an explosion in the scale and capabilities of language models.

LLMs are called "large" because they contain an enormous number of parameters - typically tens to hundreds of billions. These parameters are the numbers that define how the model processes input and generates output. Parameters in LLMs are essentially the learned values that the model uses to make predictions. They represent the knowledge the model has acquired during training. These parameters are adjusted during the training process to minimize the difference between the model's predictions and the actual correct outputs.

Some well-known LLMs and their parameter counts include:

1. GPT-3 (Generative Pre-trained Transformer 3) by OpenAI: 175 billion parameters

2. PaLM (Pathways Language Model) by Google: 540 billion parameters

3. LLaMA (Large Language Model Meta AI) by Meta: ranges from 7 billion to 70 billion parameters

A Large Language Model (LLM) with more parameters can understand and generate more nuanced and accurate responses, as it has a greater capacity to learn from diverse and complex data patterns.With more parameters, an LLM can capture a wider range of linguistic subtleties, enabling it to perform a variety of tasks with higher precision and adaptability.

How do Large Language Models Work?

Large Language Models (LLMs) use self-supervised learning, which differs from traditional supervised learning methods by relying on vast amounts of unlabeled text data instead of carefully labeled datasets. This eliminates the need for manual labeling, allowing training on much larger datasets. A labeled dataset pairs each piece of data with the correct answer or category, like an image labeled "cat" or "dog" for image recognition. Creating such datasets is time-consuming making LLMs' ability to learn from unlabeled data extremely valuable.

The core task of most LLMs is predicting the next word in a sequence. By doing this repeatedly on enormous text collections, the models learn complex language patterns and relationships, considering the entire context to understand nuances and generate coherent text.

LLMs use a neural network architecture called a transformer, designed to handle sequences of data like sentences or code. Transformers understand each word's context by considering its relationship to every other word in the sentence, allowing the model to grasp the overall structure and meaning.

During training, the model starts with random predictions for the next word in a sentence. With each iteration, it adjusts its internal parameters to improve its predictions. Over time, the model becomes skilled at generating coherent sentences and understanding language in various contexts.

Customizing LLMS

1. Fine-tuning: Updating the model's parameters using domain-specific data to improve performance on particular tasks. Imagine you have a basic recipe book with general recipes for various dishes. Fine-tuning is like adding specific instructions and tweaks to make the recipes better suited for a particular cuisine, like Italian. So, you take the general pasta recipe and refine it using authentic Italian ingredients and techniques to make it perfect for Italian cooking.

2. Pre-training: Creating new Foundation Models tailored to specific domains or modalities using substantial computational resources and unique datasets. Think of pre-training as building a brand-new, specialized encyclopedia from scratch. You gather a huge amount of information and spend a lot of time and effort organizing it specifically for a topic like space exploration. This new encyclopedia is now tailored to provide detailed and specific information about space sciences.

3. Reinforcement Learning with Human Feedback: Refining model outputs using human preferences, addressing challenges in subjective interpretation. Consider teaching a child to draw. The child makes a drawing, and you give feedback, like saying, "I like how you drew the sun, but can you make it bigger and add some more rays?" The child takes your advice and adjusts their drawing. Over time, with more feedback, the child learns to draw better.

Working with LLMS

1. Prompt Engineering: Crafting effective prompts to elicit desired responses from the model. This is often the first step in using a model for a specific task. Imagine you're at a restaurant and want a specific dish. Instead of just saying, "I want something to eat," you give a clear prompt: "Can I have a spaghetti carbonara with extra cheese and a side of garlic bread?" This specific request helps the waiter bring exactly what you want, similar to how a well-crafted prompt helps an AI model generate the desired response.

2. Retrieval Augmented Generation (RAG): Enhancing model responses by incorporating relevant information retrieved from a corpus of documents. Suppose you're writing an article about the history of the internet. Instead of relying solely on your memory, you look up and include key facts from reliable sources, like books or online articles, to make your article more accurate and informative. Similarly, RAG improves model responses by incorporating relevant information retrieved from the internet or your documents.

We will explore all these methods in detail in the upcoming tutorials.

Applications of Large Language Models

LLMs have a wide range of applications in various fields:

• Chatbots, virtual assistants and customer service

• Content generation

• Language translation

• Summarisation

• Software development (code generation and review)

Challenges and Considerations

While LLMs offer immense potential, they also come with challenges:

• The need for substantial computational resources, especially for larger models.

• The importance of choosing the right model and customisation technique for specific use cases.

• The potential for unexpected outputs, particularly when processing long or complex inputs.

Small Language Models

Small Language Models (SLMs) are a streamlined alternative to large language models like GPT-3. With fewer parameters—often under 100 million—SLMs are faster, more cost-effective, and energy-efficient, making them ideal for use in edge devices, mobile phones, and specialized tasks.

Key Advantages of SLMs:

• Speed & Efficiency: Faster inference and real-time processing.

• Cost-Effective: Easier and cheaper to train and deploy.

• Energy-Efficient: Suitable for devices with limited power, like mobile phones or IoT devices.

Examples of Popular SLMs:

• DistilBERT (OpenAI): 82 million parameters, great for tasks like text classification and sentiment analysis.

• ALBERT (Google): 12 million parameters, optimized for efficiency with fewer resources.

• ELECTRA Small (Google): 14.5 million parameters, designed for tasks requiring real-time processing.

Conclusion

We've covered the essentials of GEN AI, from Foundation Models to Large Language Models. This overview sets the stage for our journey into AI from a JavaScript developer's perspective. In our next post, we'll delve deeper into the workings of these models. We'll explore key concepts like tokens and embeddings, focusing on what's relevant for JS developers rather than diving into Machine Learning complexities.

See you in the next one!