New and Powerful Components for AI and LLMs

Shoty Ashnai, the R&D manager of Sound and Vision, warmly welcomes attendees to the final session of the Wolfram 141 webinar series. The series has covered a wide range of topics including new features, mathematical computations, compiler systems, and various data visualizations. Recordings of past sessions are available on the webinar landing page. Today's focus is on AI and LLMs, with presentations by Julio Alisand reini on prompt engineering and updates by Shoty on audio, video, and image computation. The session also includes a Q&A with Bradley Ashby and Timo Berrier. The audience is encouraged to participate in a poll and ask questions through the chat.

The script instructs participants to access version 141 of the Wolfram app for the webinar examples and mentions a free trial offer. A poll question is asked regarding the upgrade to version 141. The webinar audience and YouTube live stream viewers are encouraged to submit questions and comments through the chat. The technical staff is present to coordinate Q&A between the webinar and live stream platforms. The script also explains the webinar interface, highlighting the download link for presentation notebooks and previous session materials.

Julia, the manager of the machine learning group, discusses dynamic prompting and connecting to Large Language Models (LLMs). She provides an overview of the LLM stack functionality in Wolfram Language, including new features in version 141. The presentation covers top-level functions for interacting with LLMs, supporting functions, options, and parameters. It also explores new service connections and future updates. Julia demonstrates how to use simple strings and complex instructions for interacting with LLMs and introduces image expressions support. The session concludes with a Q&A segment.

The script explains the support for image expressions in LLM interactions, allowing for direct image inputs without the need for base64 encoding. It showcases how to start a conversation with an LLM using an image and receive generic information about the image content. The ability to mix text and images in inputs is demonstrated, highlighting the multimodal input feature. The script also discusses the internal representation of different input types and the potential for future model capabilities.

The script introduces the LM function, which allows for delayed evaluation and user input in interactions with LLMs. It demonstrates how to set up interactions and reuse them with different inputs. The use of string templates for creating parametrized inputs is explained, along with the ability to customize function behavior using the LM function's interpreter type. The script also shows how to create self-documented code using named arguments and how to mix multimodal inputs for image captioning.

The script describes how to create conversational interfaces with LLMs using chart objects and the chart evaluate function. It shows how to start a persistent conversation by modifying chart objects and preserving conversation history. The use of multimodality in conversations is demonstrated, along with the ability to export conversations as chart objects for later manipulation.

The script explains how to customize interactions with LLMs using various options and parameters. It demonstrates how to control the interaction via the LM evaluator option and how to use the llm configuration option to set multiple parameters. The script also shows how to set default configurations for the LM evaluator and how to use service connections to specify different LLM services.

The script delves into prompt engineering, explaining its importance in refining LLM interactions. It introduces the llm prompt function and the ANM prompt repository, which contains well-crafted prompts. The script shows how to use prompts to simplify answers, change languages, and integrate tools for live information retrieval. It also demonstrates how to use tool prompting to enhance LLM responses with external tools and how to create new documents with plots and other information.

The script introduces retrieval-augmented generation (RAG) in LLMs, which allows for on-the-fly argumentation of prompts based on conversation context. It demonstrates how to use the LM prompt generator to enrich prompts with contextual information. The script also explains the use of semantic search and vector database functionality to provide relevant context to LLMs based on the semantic sense of the input text.

The script provides an overview of vector databases and their use in finding similar content numerically. It explains how to create and search vector databases and how to use feature extractors to convert data into vectors for searching. The script also discusses the use of semantic search to convert text into vectors that represent semantic content and how to perform semantic searches using indexes created from various text sources.

Shi Ashnai discusses updates in video, audio, and image computation in Wolfram Language version 141. She highlights improvements in video quality of life, support for the latest FFMpeg version, and the addition of image resolution options for videos. The script also covers new functions like video summary plot for quick content display and updates to subtitle handling. The presentation includes demonstrations of video generation functions and the new manipulate video function, which allows for detailed control over variable changes over time in video creation.

The script explains advanced video manipulation techniques introduced in Wolfram Language version 141. It covers the use of key frame actions in manipulate and manipulate video functions, allowing for precise control over how variables change over time. The script also introduces video frame fold, which operates similarly to fold list for video frames, and the use of sew video and reap video for conditional frame extraction. The presentation includes examples of how these functions can be used to create motion blur, remove backgrounds, and perform other video manipulations.

The script highlights new video generation functions in version 141, such as constant video from a single image and video transcribe, which adds transcribed subtitles to video objects. It also discusses enhancements to speech recognition with the introduction of the Whisper model, which significantly improves recognition speed and accuracy. The script covers updates to subtitle handling, including the ability to extract subtitled strings and rules, and the addition of language identification and translation capabilities for video objects.

The script discusses updates in image computation, particularly the upgrade of OCR functions on Mac OS to utilize operating system APIs, improving text recognition in various scenarios. It also covers the introduction of people segmentation in the remove background function, which can be applied to both single images and video frames to target and remove backgrounds effectively.

The script concludes the webinar with closing remarks, thanking attendees and the behind-the-scenes staff. It mentions the availability of a survey for feedback and certificates of attendance. The presenters encourage participants to explore and provide feedback on the new features introduced in Wolfram Language version 141. They also address common questions from the Q&A chat, summarizing key points and providing additional insights on vector-based functionality and semantic search.