Define the workload.
Agree the task, models, tools, data permissions, expected usage, region and timing. List what is known and what still needs checking.
What to recordA workload brief and a list of open questions.
PILOT PROCESSAgree the test.
Agree the test.
Then judge the result.
Start with one workload and the setup you use today. A pilot should produce enough evidence to decide whether the next step is worthwhile.
Set the comparison.
Choose representative tasks and pass criteria before testing. Include difficult cases, incorrect actions and the points where a person needs to intervene.
What to recordAn evaluation set, baseline and test plan.
Review the results.
Compare quality, performance, reliability and the cost of a completed task. Investigate failures and decide which changes deserve another test.
What to recordResults, limitations and the proposed next step.
Agree deployment.
If the results support moving ahead, confirm capacity, pricing, timing, responsibilities and support in a proposal. No fixed delivery schedule is implied by this process.
What to recordA deployment proposal and acceptance requirements.
FOUR THINGS TO MEASURELook at
Look at
the whole task.
A fast model call is one part of the result. Evaluate the application, its tools and the work needed to keep it running.
Quality
Completed tasks, incorrect actions and human intervention.
Performance
Response time, throughput and concurrency across the workflow.
Reliability
Failed calls, recovery, retries and retained task state.
Cost
Model APIs, compute, storage, networking and operating support.
LET'S START WITH THE WORKWhat do you
What do you
need to run?
Tell us about your application, the models you use and the capacity you need. Estimates and open questions are welcome.
Prepare a workload brief ↗Non-sensitive information first. Current customer outreach focuses on US customers.