NVIDIA Launches Locally Run AI Chatbot, Chat with RTX

The release of OpenAI’s ChatGPT prompted creators and companies to build their own large language models (LLM) and chatbots. It has also become common to see companies utilize open-source models to create AI-powered products and services.

While numerous LLMs like Claude, Llama, Cohere, and so on, have been made and used to create chatbots after ChatGPT’s success, none have had the ability to run locally on one’s computer, always requiring internet and cloud connection.

Running a game-changing move, NVIDIA recently developed and launched a demo app named Chat with RTX, equipped with GPT-like models that have the ability to process one’s local data and files of various formats, completing queries and answering questions based on information extracted from them.

Local and Personalized AI

Similar, though not completely the same, to ChatGPT, Chat with RTX lets users ask questions or provide queries for the LLM to complete. While ChatGPT’s services revolve around more broad and comprehensive information, as the model used is trained on numerous datasets from the internet, Chat with RTX’s abilities are limited to the data users provide it.

Usable locally on one’s PC, Chat with RTX performs tasks based on one’s personal datasets, providing tailored responses according to the information that users upload for it to extract and analyze. Chat with RTX can process data in the form of various formats, such as text, doc or docx, PDFs, and XML.

Chat with RTX’s features are perceived as valuable by most of its current users, easily boosting productivity as they allow users to no longer have to waste time and energy searching through mountains of notes and information, instead making them a dataset that Chat with RTX can swiftly explore as well as learn and get answers from with a snap of its hypothetical finger.

Aside from local data, Chat with RTX also has the ability to perform queries based on YouTube URLs. Users can input the link to YouTube videos or playlists, which the app will then load and scan to get transcriptions of said videos.

Since the LLM app runs locally, however, Chat with RTX can only access said transcriptions and not what the video presents visually. For instance, users can ask questions regarding what a person says in the video but cannot know what color shirt the person is wearing if the information isn’t audibly stated.

Speed and Security

Chat with RTX’s locality, courtesy of NVIDIA’s retrieval-augmented generation (RAG), TensorRT-LLM software, and RTX acceleration, allows it to perform and provide results fast, especially compared to cloud-based models and LLM services that require an active connection.

By performing locally, Chat with RTX can also provide users with more privacy and security. Since all the data provided to the app already exists on one’s PC and will continue to stay that way, any sensitive data or information will not be shared with a third party and will not be exposed to, or at the very least, be less vulnerable to malicious online viruses.

With it being uncommon for early apps and LLMs to still need some fine-tuning, especially considering that this is only NVIDIA’s demo app, perhaps Chat with RTX is simply a diamond in the rough, with the potential of someday outperforming existing LLMs out there.

Those interested in trying the demo app can download Chat with RTX for free through NVIDIA’s website. It is crucial to note that Chat with RTX is only available and usable for PCs running on NVIDIA GeForce RTX 30 Series GPU or higher, with at least 8 gigabytes of VRAM. To use the app, users will also have to download Mistral and/or Llama 2.

Source: NVIDIA Announcement

Meme & AI-Generated Picture

Job Posting

• NVIDIA - Principal Hardware Systems Architect - Santa Clara, CA (Remote/Hybrid)

• NVIDIA - Senior Software Engineer, Fabric Networking, GPU - Santa Clara, CA (Remote/Hybrid)

• NVIDIA - Senior System Software Engineer, DriveSim Content API - San Francisco, CA (Remote/Hybrid)

• NVIDIA - Technical Marketing Engineer, Enterprise Platforms - San Francisco, CA (Remote/Hybrid)

Promote your product/service to Digger Insights’ Community

Advertise with Digger Insights. Digger Insights’ Miners are professionals and business owners with diverse Industry backgrounds who are looking for interesting and helpful tools, products, services, jobs, events, apps, and books. Email us [email protected]

Your feedback would be greatly appreciated. Send it to [email protected]