A good evaluation turns “this looks promising” into a decision you can explain and repeat.
Collect representative work
Begin with examples of the task you expect the system to perform. Include ordinary cases, incomplete information, ambiguous requests, and cases that should be handed to a person. Use examples you are authorised to work with and remove information that is unnecessary for the test. Keep a separate set for evaluation so the same examples do not become both your design target and your proof of success.
Write down the judgement
Describe what makes an answer useful. A support response might need to cite the right policy, address the actual question, and avoid promising something the organisation cannot deliver. Separate those criteria so a fluent answer cannot hide a factual mistake. For subjective work, give reviewers examples and discuss disagreements before turning their scores into a release decision.
Test the whole workflow
Evaluate more than the final text. Check whether the system retrieves appropriate information, uses permitted tools, asks for clarification when needed, and recovers from a failed dependency. Record time and cost alongside quality. A strong result in isolation may still be awkward or expensive in the process where people use it.
Keep the evidence attached
Record the model, instructions, source material, and relevant settings with each run. Revisit important examples when those inputs change. Evaluation is most useful when it makes a release decision clearer and helps the team locate what needs to improve.
