Frequently asked questions
What Seven Eighths Lab does, how it builds benchmarks, and how companies can work with it.
What is Seven Eighths Lab?
Seven Eighths Lab studies how to turn the software, systems and workflows that actually exist inside companies into reproducible AI benchmarks and reinforcement-learning (RL) environments. The lab builds benchmarks from real work, evaluating how well AI agents complete enterprise tasks in executable environments, against verifiable criteria. It is an initiative of Knowlyr, and the site is operated by Shanghai Knowlyr Technology Co., Ltd.
Why is it called Seven Eighths?
The name comes from the iceberg: only one eighth shows above the water. Public benchmarks and exam-style questions are the visible eighth, and AI scores there are nearly perfect. What decides whether AI delivers value at work lies below the surface — long chains of tasks, processes no one wrote down and the judgement held by experts — and that is what the lab turns into benchmarks.
How is Seven Eighths Lab related to Knowlyr?
Seven Eighths Lab is an initiative of Knowlyr. The site is operated by Shanghai Knowlyr Technology Co., Ltd. The lab’s open-source tools are published on Knowlyr Open Source (knowlyr.com/en/opensource).
Why are public benchmarks getting worse at telling models apart?
Because many of them are saturating: leading models score close to one another and near the ceiling, so the benchmark stops telling us much that is new. We see at least three causes: task distributions are detached from real work; once tasks are public they may enter model corpora, so higher scores do not necessarily mean higher capability; and pass/fail outcome scoring is too coarse to tell a run that was on track but unfinished from one that went wrong from the start.
How do enterprise task benchmarks differ from public benchmarks?
Enterprise task benchmarks draw their tasks directly from work that actually exists, rather than from questions written by researchers or collected from public competitions. They cover workspace and software-engineering tasks from real operations, with executable checks and scoring criteria. Real work is full of incomplete requirements, legacy code and cross-system dependencies — exactly what public benchmarks tend to miss.
What does Seven Eighths Lab research?
The lab works in three areas: enterprise task benchmarks, RL environments and evaluation methods. RL environments wrap legacy systems and business software as interactive, resettable sandboxes for long-horizon agent evaluation. Evaluation methods cover difficulty grading, contamination control, reproducible scoring and expert-designed rubrics.
Has the lab released any benchmarks yet?
Not yet — none of the three areas has been released, and the research note states there are no experimental results so far. When the lab does release, it will report only results it has run itself and publish the evaluation protocol. Progress will be posted on the Benchmark page and in the research note.
How do you turn a legacy enterprise system into an RL environment?
By mocking it so it runs fully inside an isolated container and can be reset every episode. The five steps are boundary identification (mapping databases, external APIs, authentication and file storage), content replacement (structurally consistent synthetic content with no personal information or trade secrets), dependency stubbing, resettable state (every episode starts from the same snapshot) and executable verification (programmatically checkable completion conditions, plus expert rubrics where needed). The result is an environment with a uniform reset–act–observe interface, usable both for evaluation and for reinforcement learning.
How do you deal with benchmark contamination?
Contamination cannot be eliminated, but it can be managed. The lab plans to separate a small public subset from a private test set used for the main evaluation; release benchmarks in versions and rotate tasks regularly; record each task’s source type and build date so results can be traced; and add step-level process scoring to reduce inflated scores from memorised answers.
What open questions is the lab working on?
The lab lists three questions it aims to answer next. How much of a real system’s difficulty does mocking preserve? How should process scoring balance expert judgement against reproducibility? How can a private test set stay credible while remaining open to external scrutiny? The lab welcomes discussion with others working on them.
Can my company’s know-how become an AI benchmark?
Start with the free online estimate: it takes about a minute and shows a reference range after a few questions. It looks at whether your business systems, workflows and expert know-how could become AI benchmarks. The result is for reference only and is confirmed after verification.
How can a company work with the lab?
There are two ways to transfer: full transfer and workflow transfer. A full transfer covers business systems and documents; a workflow transfer covers key workflows and the hard cases within them, after which your expert walks the lab through them. You decide what to share, and the scope goes into the contract.
Does working with the lab cost anything?
No. The lab charges companies nothing — no fees for the estimate, the verification or the contract. Rights and obligations are governed by the signed contract.
How are the materials we share used and protected?
They are used only to build benchmarks, within the scope of the contract; the lab takes usage rights only and claims no ownership of the originals. Personal information and trade secrets are removed before delivery, and originals are handled in isolated environments wherever feasible. The contract prohibits reselling the original materials or re-identifying their source, and software ships with a software bill of materials (SBOM) for compliance review.
Will you disclose our company’s identity?
No, not unless you agree in writing. The lab never discloses a partner company’s identity, and the contract prohibits re-identifying the source of the materials.
Who can see the contact details I leave?
Only Knowlyr. Your phone number or WeChat is never given to anyone outside. Locking a result takes just one way to reach you; the result and its reference are kept for 3 days, after which you get a new estimate.
Can we step out before signing?
Yes — until you sign, you can step out at any point. Just tell the lab and it will stop contacting you. Rights and obligations are governed by the signed contract.
How can AI researchers get involved?
Reach the lab through Knowlyr’s contact page; the open questions in the research note are open for discussion. The lab’s tools are on Knowlyr Open Source, and its methods are set out in the research note “Benchmarks from real work: saturation, mocks, contamination”. Knowlyr is also hiring — see its Careers page.
Answers summarise this site’s pages; the signed contract prevails. See also: About · Benchmark · Research note · The rules, up front · Estimate