Rova AI’s LLM Analytics and Evaluation Platform

Large language models have become as significant as ever in our modern, tech-filled world, with more and more expansive use cases seen every day. With how diverse our society’s needs have increasingly become, fine-tuning LLMs, which is to customize and mold pre-trained models into specialized ones to perform in particular domains, is now widespread.

Once created, LLMs can be further trained and fine-tuned, often with the main goal of improving their performance. This can be done by conducting evaluations and identifying aspects to maintain or remove when reading and reviewing chat logs. Evaluation frameworks can also be used to assess generalization, scalability, reasoning, and so on. However, some parties believe that, in reviewing and improving LLM performance, what matters most is conducting comprehensive and in-depth user interaction analysis to determine the value of models.

This includes YC-backed startup Rova AI, founded by Avigya Basnet, Karan Bhasin, Samarth Kadaba, and Josh Sanyal, which has created a platform that analyzes chats between users and LLMs to see how user behavior influences KPIs to automatically curate evaluations and fine-tunings.

User Behavior and Feedback

KPIs, otherwise known as key performance indicators, are quantifiable measurements that can be used to gauge a specific asset’s performance. Though KPIs are generally seen through revenue growth, revenue per client, or profit margins, in the case of LLMs, KPIs include customer retention, conversion, and satisfaction when using one’s model.

Rova AI’s founders’ experience in building LLM-powered copilots for financial analysts led to them finding out the incredibly complex role user behavior plays in model performance evaluation and, thus, in software development. This discovery prompted the quartet to infer a model’s performance by utilizing all the product signals prompted by user interaction.

Rova AI believes that the ever-changing behavior of users makes evaluation sets, which are typically generalized, ineffectual. As a result, the startup’s platform pays attention to certain moments and interactions, such as the times when users express frustration toward the answer they receive, for example, or the circumstances in which users come up with follow-up prompts, presumably due to dissatisfaction.

These interactions are then analyzed to help differentiate between good and bad signals more efficiently. As a result, the platform devises more comprehensive user trends. When traditional methods of tracking user behavior require creators and trainers to excruciatingly look through mountains of chat logs, Rova AI lets users quickly search for what they need using chat topics or semantics.

Performance Evaluation

Keeping in mind that training models on the wrong data can negatively impact model performance, most times in quite a severe manner, Rova AI wants to ensure that its users get to improve their models the best way possible.

Once user behavior is thoroughly known and observed, all the chat inputs, LLM outputs, and analyzed product data can then be exported. Successful user interactions can be used to fine-tune models, and less-than-ideal performances can be integrated into the model’s evaluation sets.

By focusing on who Rova AI’s team trusts is the center of all LLMs, that is, the user and what they value, the startup believes that the most relevant and high-quality data for finer performance can be curated and utilized for fine-tuned production. When, at the end of the day, conversions and retention rates are of significance, what better than to listen to the feedback of those behind them?

Those interested in trying out Rova AI’s platform can book a demo by contacting the startup’s team via email or request early access to the product through the website’s login page.

Meme & AI-Generated Picture

Job Posting

• Block - Technical Program Manager, Sales & Account Management - San Francisco, CA+ (Remote/Hybrid)

• Cisco Meraki - Senior Product Designer, Secure Connect & Meraki Security - New York City, NY+ (Remote/Hybrid)

• Takeda - Principal Electronic Data Capture Engineer - Los Angeles, CA+ (Remote/Hybrid)

• UL Solutions - Senior Auditor CRS - Seattle, WA+ (Remote/Hybrid)

Promote your product/service to Digger Insights’ Community

Advertise with Digger Insights. Digger Insights’ Miners are professionals and business owners with diverse Industry backgrounds who are looking for interesting and helpful tools, products, services, jobs, events, apps, and books. Email us [email protected]

Your feedback would be greatly appreciated. Send it to [email protected]