Article
On-premises AI: what to test in a pilot
Suppose employees spend part of each day looking up instructions and explaining them to colleagues. You want AI to take on some of that work while keeping the documents inside your company. A pilot needs to answer three questions: can the model do the job, how long does it take to check its answers, and who will maintain the system?
Start with one task
Choose a task with a result you can check. An assistant might find an answer in an instruction document and show the relevant passage. Or it might extract fields from an incoming document for an employee to compare with the original. In either case, the input, expected output and possible mistakes are clear.
We will use an assistant for internal instructions as our example. This is a way to plan a pilot. At Avenex, we start by discussing one task and test the solution on the company’s actual data.
Collect documents and real questions
Gather the documents employees currently use, along with real questions and answers written by a specialist, with references to the relevant passages. Set some examples aside for testing after configuration. They will show how the assistant handles questions the developer has not already worked through.
Include awkward cases too: an old instruction alongside a new one, a question missing key details, or a document that contains no answer. These make it easier to spot confident statements that the source does not support.
- An ordinary question: the answer appears in the current instructions.
- Missing details: the assistant needs to ask a follow-up question.
- No answer: the assistant should say so plainly.
- An outdated document: check which version the answer cites.
Agree on what counts as a good result
Decide how to assess the answers before starting the pilot. A specialist checks whether each answer is correct, whether its reference supports it, and how much manual correction it needs. Record serious errors separately: an invented rule, a wrong deadline or a reference to the wrong document can make an answer unusable.
Compare the whole process with the employee’s usual workflow: searching, reading, answering and checking. Fast generation helps only if corrections do not use up all the time saved. Set a quality threshold for your own task; there is no universal percentage of correct answers.
- The question and the document used to check the answer.
- The assistant’s answer and its source reference.
- What the specialist corrected or rejected.
- The time needed to obtain and check the answer.
Decide where data stays and who can access it
If documents must stay inside the company, write that requirement down together with a list of permitted external connections. Check the entire data path: document uploads, processing, conversation history and error logs. Calling a system ‘local’ does not settle these questions.
Clarify access rights: which instructions each employee may read and which answers they may receive. During the pilot, leave sending emails, changing records and other operational actions to people. With Avenex, you can discuss deployment within the client’s network and handover to its IT team.
Decide what happens after the pilot
Review the errors, the time spent checking answers and the cost of running the system. If the results meet your needs, decide who will update documents, check answers after a model change and investigate employee reports. If too much correction is needed, narrow the task or postpone deployment.
A one-page pilot report is often enough: what you tested, which examples you used, where the system failed and what you decided to do next. It gives the team a clear basis for the next step.
Discuss your project
For an initial conversation, bring a description of the task, a few sample documents you are allowed to share, and your data restrictions. Explain how employees do the work today and what they want to make easier. That is enough to start discussing a pilot with Avenex.
Discuss a pilot →