Superhuman, the AI productivity platform, delivers AI to tens of millions of people wherever they work. The company started as the Grammarly AI writing assistant; today, its products also include Superhuman Mail, Superhuman Docs (formerly Coda), and Superhuman Go, a new proactive AI assistant that includes a growing set of AI agents. Every one of Superhuman’s product features has to meet a quality bar before it reaches a user – and the more the platform grows, the harder that bar is to maintain.
That work falls to Superhuman's evaluation pipeline, which compares model versions, scores generated suggestions against quality criteria, and routes outputs through human review before they go live. For years, that pipeline ran on tooling the team had built in-house – which worked well when evaluations were fewer and more uniform, but the recent expansion changed the math. More evaluations, across more formats and more languages, were arriving faster than a homegrown system could absorb. That challenge led Superhuman to partner with SuperAnnotate to rebuild the pipeline for the scale the platform had reached.
The Challenge
A few years ago, annotation at Superhuman (then Grammarly) meant collecting large training datasets. Today, it means evaluation: comparing different model versions, rating generated suggestions against quality criteria, and running human-in-the-loop validation on LLM outputs that ship to users. With the recent acquisitions of Superhuman Mail and Coda (now Superhuman Docs), and a strategic shift toward agentic, multilingual, and enterprise-search features, the variety and volume of evaluations grew sharply.
Superhuman’s evaluations have two non-negotiable properties:
- They have to look like Superhuman’s products. Reviewers don't just rate words – they evaluate suggestion placement, underline color, the card that pops up when a user hovers over an alert, and other product visual specifics. The annotation UI has to render the real product experience so that quality teams are evaluating what users will actually see.
- They have to move fast. Evaluations have a 24-hour SLA for internal users testing new features. Datasets are typically hundreds of items, refreshed multiple times a week. That cadence leaves no room for tooling friction.
Anything in the evaluation stack that couldn't keep up with this – speed, fidelity, or flexibility – would become a bottleneck on Superhuman’s broader AI roadmap.
Maintaining and developing our internal tool used engineering resources that could be better leveraged on other work. Moving to a high-quality third-party platform allows us to redirect that time and also benefit from a team of dedicated developers.
Previous Solution and Key Challenges
The team had been running on a homegrown annotation platform, which fed off a quality-evaluation orchestration system that generated evaluation tasks programmatically and shipped them in for human review. A separate vendor handled multilingual and crowd-scaled work.
The setup worked, but it was straining under the team's ambitions:
- A capability-shaped bottleneck. Job creation and the UI relied on a custom HTML-based markup language that required deep familiarity to build or modify safely. Non-engineers – linguists, data scientists, project admins – couldn't self-serve.
- No native ML pipeline integration. The team’s internal tool had no built-in workflows, no concept of multi-stage projects, out-of-the-box quality control, or model-in-the-loop functionality. Pipelines were stitched together manually with scripts and CSV exports.
- A growing build-vs-buy tax. With Superhuman’s AI roadmap accelerating, every engineering hour spent maintaining an internal annotation platform was an hour not spent on Superhuman’s core product. The team had a backlog of internal tooling features it would never realistically ship.
- No path to LLM-as-judge. The team's stated direction was to have LLMs handle a majority of the annotation pipeline within a year, with humans correcting and validating. The existing setup had no framework to support that pattern.
What Superhuman was looking for in a replacement was specific:
- A platform flexible enough to embed the team's own UI – including the custom JavaScript library and overlays the team had invested in – so quality teams could keep evaluating real product experiences.
- A platform that non-engineers could configure for new evaluation use cases without filing a ticket.
- Native multi-stage workflows, QA, and automation so the team would stop rebuilding these from scratch.
- A roadmap and architecture that supports LLM-as-judge with human-in-the-loop, not just classic labeling.
- Enterprise-grade security: SSO enforcement, SOC 2, ISO 27001, audit logging.
- A partner willing to prioritize Superhuman’s requests as the team's needs evolve through acquisitions.
The Solution
After evaluating multiple SaaS annotation platforms against a structured scorecard, Superhuman chose SuperAnnotate. Three things tipped the decision:
1. A customizable UI layer that didn't force the team to give up its own components.
During the pilot, SuperAnnotate's custom solutions team built a web component wrapper around the team's existing Portable UI, letting them carry over their existing JavaScript components – including the side-by-side suggestion comparison UI that mirrors the product. At the same time, SuperAnnotate's drag-and-drop multimodal editor gave non-engineers a path to spin up new evaluation forms (radio buttons, ratings, span labeling, conditional logic) without writing code. Superhuman didn't have to choose between fidelity and accessibility – they got both.
2. A complete evaluation operating layer, not just a labeling tool. SuperAnnotate provided several key native features:
- Multi-stage custom workflows with annotator → QA → audit paths and configurable statuses.
- An Orchestrate pipeline engine where Python actions can be triggered by status changes – to run LLM-as-judge, calculate consensus, route low-confidence items to human review, or send completed batches back to the team’s internal orchestration system.
- Quality assurance built in: sampling-based QA (e.g., 20% routed to QA automatically), gold/honeypot items, scoring metrics tracked per annotator across projects, return-with-comments flows, and Krippendorff's alpha calculation at the project level.
- Native annotator management: roles, permissions, automatic task assignment, flag-and-reassign for out-of-expertise items, and item-level traceability of who saw and rated what.
- Native integrations with the cloud and data-platform tools in use across the organization – eliminating the m
- anual CSV shuffle the team had been doing.
3. A migration path that respected Superhuman’s reality. Rather than asking the team to redo everything from scratch, the SuperAnnotate team mapped the migration in three waves aligned to actual usage:
- Automated internal-system evaluations (~90–95% of volume) – automated via the SuperAnnotate Python SDK, with
- Superhuman’s internal system creating folders, uploading data, assigning annotators, and pulling results programmatically.
- Custom one-off evaluation jobs – migrated onto Sup
- erAnnotate's multimodal editor and templates.
- Multilingual / crowd work currently on the external vendor – migrated later, once the core pipeline was stable.
SuperAnnotate also committed to a path for Superhuman
’s robust enterprise security requirements: SSO enforcement, weekly authentication audit logs as an interim measure, and 180-day audit log retention – alongside the existing SOC 2 and ISO 27001 posture.
Two dominant factors carried the most weight in our decision to move forward with SuperAnnotate. The platform's flexibility was key to achieving the look and feel of the evaluation UI we needed, and the support we received during the pilot not only made implementation possible but also gave us confidence that they would be good partners moving forward.
Results
Within the pilot and the first phase of the rollout, the partnership delivered concrete wins:
- Enabled more self-serve annotation job creation, removing engineering as a bottleneck. New evaluation use cases that were previously gated on engineering are now launched self-serve through the multimodal editor for the majority of use cases.
- Returned ~20 hours of engineering time per week. Engineering resources previously spent maintaining the internal tool were redeployed to Superhuman's core product roadmap, with time savings continuing to compound as workflows are modified and new functionality is added.
- Maintained the 24-hour SLA on internal evaluations throughout the migration, with no disruption to evaluation turnaround.
- Annotator agreement held flat or improved across migrated projects, as measured by Krippendorff's alpha and adjacent-agreement metrics – meeting Superhuman's explicit "no quality regression" success criterion.
At the end of the day, it all comes down to our ability to confidently ship AI features and advance our mission to ‘unlock the Superhuman potential in everyone.’ Knowing we could add new functionality without compromising quality or efficiency was the deciding factor. SuperAnnotate gave us a path to scale evaluations faster, involve more of the team directly, and keep the fidelity of the product experience intact.
Looking Ahead
With the core evaluation pipeline running between Superhuman’s internal system and SuperAnnotate, Superhuman is now scaling into the next phases of the partnership:
- Migrating multilingual and crowd-scale work off the external vendor and onto SuperAnnotate.
- Expanding LLM-as-judge from a handful of evaluation types to the majority of Superhuman’s evaluation surface, with humans reserved for the high-leverage cases.
- Supporting the agentic feature roadmap – multi-turn, multilingual, and enterprise-search evaluations – on the same shared platform.
For Superhuman, evaluation has stopped being a bottleneck on its AI roadmap. It's become the multiplier behind it.
About Superhuman
Superhuman (formerly Grammarly) is the AI productivity platform on a mission to unlock the superhuman potential in everyone. The Superhuman suite of apps and agents brings AI wherever people work, integrating with over 1 million applications and websites. The company’s products include Grammarly’s writing assistance, Docs’ connected surface for teams, Mail’s inbox management, and Go, the proactive AI assistant that understands context and delivers help automatically. Founded in 2009, Superhuman empowers over 40 million people, 50,000 organizations, and 3,000 educational institutions worldwide to eliminate busywork and focus on what matters.
About SuperAnnotate
SuperAnnotate is the enterprise AI data platform that helps the world's leading AI teams build efficient human data and evaluation pipelines to ship better agentic, multimodal, and frontier AI. The platform combines flexible annotation and evaluation tooling with human and agent-in-the-loop workflows, advanced quality control, and the option to tap into SuperAnnotate's expert talent network when teams need to scale beyond their internal capacity. SuperAnnotate is trusted by frontier AI labs and applied AI teams around the world, and is backed by NVIDIA.
Want to see how SuperAnnotate could work for your team? Book a demo.