Ahead of Investing in an ML Project - A Case Study
Curious what value a properly run ML workshop can deliver and how it leads straight to smarter investment decisions? This article walks you through a complete example from real life.
A growing number of companies are starting to look at machine learning – to automate more of their processes, to improve the user experience, or simply to keep pace with competitors. An idea appears – sometimes sparked by a conference, a chat with the tech team, or a market signal that “others already have something with AI, and we don’t yet.”
That’s when many feel tempted to go straight to execution: hire a team, choose a tool, start training something – just to “see how it goes.” Having worked on machine learning projects for years, I see that as a direct route to frustration and a wasted budget.
Machine learning is not a plug-and-play technology. How well it works depends not only on the model but on the whole environment: the data, the processes, the integrations and, most of all, a correctly defined problem. This is why a machine learning workshop is not a formality but a critical project phase.
The workshop is where expectations collide with reality:
- We check the idea against the real historical data the company has (or lacks)
- We confirm whether the problem the company wants to tackle makes sense from an ML standpoint
- We decide how success will be measured, both technically and in business terms
- We propose other ways of reaching the project goal and an estimated budget; sometimes simpler, cheaper or more scalable ones
- We define scope, timeline, and costs - so you can assess ROI and check whether the project fits your budget
This post describes how we ran a custom machine learning workshop for a client developing a data extraction platform for accounting professionals.
The first part introduces the client’s business context: who they are, what problem they wanted to solve and why they contacted us. After that, I explain how we planned and prepared the workshop, from designing the template to drafting a realistic project timeline.
Next, we show how the workshop went in practice: which data we looked at, what technical and business constraints came up and what we learned about the users and the platform. Lastly, we describe how all of this shaped the final proposal, including cost estimates, success metrics and a roadmap ready for deployment.
Curious what value a properly run ML workshop can deliver and how it leads straight to smarter investment decisions? This article walks you through a complete example from real life.
It’s a long read, so make yourself a coffee and reserve around 15 minutes. That’s a small investment, but well worth it.
Business context: client, need, technology
The client
A company building a platform for accounting professionals got in touch with us. The platform’s main value is automating how data is extracted from invoices and passed on to ERP systems or finance management platforms.
How did they find us?
The first contact came via one of our developer teams that had spent several months working on the platform’s backend. Their work was appreciated, which led to a bigger team and wider responsibilities. Happy with the cooperation, the client chose to draw on our artificial intelligence expertise to evaluate feasibility and potential ROI.
The problem to solve
The client had plenty of ideas for applying machine learning but had never started any initiatives in this area before. The most urgent problem they pointed to was extracting data from invoices. The existing legacy AI system handles documents in only one language, whereas adding multilingual support is a key business requirement.
An example of a system that extracts data from semi-structured documents. Source: Microsoft.
The legacy system is hard to maintain, and even small improvements take considerable effort. The aim is to replace it with a new solution built on modern machine learning technology.
For a machine learning engineer, this is a textbook case of key information extraction from visually rich documents. Two years ago, I worked on a similar project for a different client, so I was already familiar with the situation and its constraints.
How we prepared for this ML workshop?
Questions
Before the workshop, I put together a list of questions for the client. They were designed so that answering all of them would address about 50% of the workshop’s goals. Realistically, though, most clients can’t give full answers in advance - and that’s okay. The list was meant to trigger internal discussion and reflection before we met.
Data presentation
I asked the client to get a few representative samples ready to show the problem we wanted to tackle. This is typically the most engaging part of the workshop and the one that yields the most insight – and frequently also the one that takes the longest.
Key stakeholders
Finally, I suggested which people I’d need from the client’s side to run the workshop. Above all, a key project stakeholder with decision-making authority who understands, from the business perspective, the limitations of and reasons for introducing artificial intelligence.
Just as important is someone who knows the platform in its current form and understands how data moves through the system, what happens to it, and what enters and leaves it. And finally, someone who understands how the legacy document processing system works.
How to prepare for a machine learning workshop?
If you intend to work with an external ML team, what you get out of the workshop will depend largely on how well you prepare. These are the four key areas worth considering beforehand:
1. Define your budget and cost expectations
AI projects need clear financial limits, covering both the initial investment and the long-term operating costs.
- What budget do you have for prototyping or an MVP?
- What OPEX level can you sustain once the system runs in production?
Having this clear is vital for choosing the right architecture, be it SaaS, a managed service or a custom in-house solution.
Stay realistic: artificial intelligence is expensive. Custom ML solutions carry infrastructure, engineering and maintenance costs that can grow fast. Without a clear business case, the investment won’t pay for itself.
AI should be introduced because it addresses a real problem, not because it’s fashionable.
Begin by checking:
- What concrete impact will it have on the business?
- What savings in cost or time do you expect?
- Does it remove a bottleneck, make scaling possible or open up new capabilities?
Once you have these answers, allocating resources is no longer guesswork; it turns into ROI planning.
2. Define what a “good performance” means to you
Machine learning is never 100% accurate. So what accuracy is sufficient for your product?
- Is 99% precision a must, or would 80–85% be acceptable?
- How expensive is a model error in your business context?
- Which hurts more – false positives or false negatives?
Clear performance expectations help match technical solutions to business priorities, particularly where trade-offs come into play (e.g., between cost, speed and accuracy). It’s a step towards planning risk management.
3. Identify technical and business constraints
Certain constraints can eliminate whole categories of solutions, so it’s important to bring them up early.
- Do you need responses in real time?
- Do you work in a multilingual environment (e.g., Europe, Asia) and need support for languages other than English?
- Do regulatory or security requirements apply (e.g., on-premise deployment, GDPR, ISO)?
Being open about these limitations from the start helps the ML team concentrate on the right design space.
4. Prepare your product context and sample data
To make the workshop actionable, get ready a simple demo or walkthrough showing how your platform works.
- Where will the ML model fit into your product’s workflow?
- Which real-world data examples, user actions or decisions would benefit from ML?
Sharing sample data points (anonymized, if needed) and presenting the product in context helps reveal edge cases, integration constraints and user experience aspects that influence the design of the solution’s design.
Project scoping: from raw data to data-driven decisions
The workshop, part of the project planning phase, took place at the client’s office. As soon as we arrived, we got down to business.
1. Kick-off
I opened with an overview of how ML projects differ from conventional IT initiatives. I described what sets them apart, what to watch for before and during implementation, and how our methodology works - from the workshop all the way to delivery. Then I took the team through how our engagements usually unfold.
2. Sample data
Following the introduction, we turned to data analysis. The client’s team presented sample documents and explained what wasn’t working and what they expected from the machine learning model. In my mind, I was already noticing hidden patterns and edge cases I had seen in comparable projects – the subtle details that are easy to miss yet can decide whether a machine learning system succeeds or fails. During the Q&A, I gathered further requirements and noted them on the Miro board to synthesize later.
3. Data volume
In the workshop, we talked about how data moves through the system: how many documents are processed, how and where the original data is kept, and who manages and maintains it. We also established where the data can be accessed, which teams or roles own it, and what that means for training, monitoring and maintaining a machine learning solution. These discussions were key to pinpointing integration points, possible bottlenecks and data governance requirements.
The monthly volume of data was considerable – several million new documents, far more than needed to fine-tune a robust machine learning model. The data lives in the client’s cloud environment and is fully accessible, so we don’t expect any problems exporting it for training, evaluation or integration.
4. Platform insights
After getting a sense of the input data, we moved on to the platform. That’s where we discovered something crucial: the client already had a quality control mechanism in place, in which users check and correct the model’s predictions. These users – accountants – are highly motivated to keep the data accurate, as their work (and liability) relies on it.
For machine learning, this is perfect. Clean, labeled training data comes in every month thanks to the regular invoice cycles, and the corrections are made by domain experts who are legally obliged to be accurate. We could safely assume that the labeled data was of high quality.
5. User challenges
The platform demo also led to a useful discussion about what matters most to users and what frustrates them. Watching how they work gave us a feel for their daily workflow. From their stories, we found out what slows them down, what they value and where automation still isn’t good enough.
For these users, the key metric is the time they spend fixing model errors by hand. Better data extraction cuts down that manual work and thus improves the platform’s UX. That’s a well-defined project goal.
6. Success metric
Each model error means extra manual work – an obvious, measurable pain point. So rather than searching for abstract KPIs, we concentrated on a direct metric: the number of errors per document or field class. It’s simple, quantifiable and tied directly to the business result.
7. Client involvement
Several times, the conversation broadened into a wider machine learning strategy - what else the client hoped to build and how they imagined scaling. They were very engaged, stressing a long-term partnership and their intention to develop and maintain the platform cost-effectively for years to come.
They also arrived with tough questions, such as suggestions to use tools like LLMs, and expected us to assess the options critically. I committed to preparing a comparison of various solution paths, covering development effort, operating costs, maintainability, retraining strategy and scalability. They also asked for our clear recommendation on what would suit their case best.
8. Business process mapping
We then mapped the process, from the moment a document enters the system to the moment the extracted fields leave it. Here, the pipeline was fairly simple - a standalone service with no downstream systems depending on its output in real time.
9. Functional requirements
After a quick lunch break, we picked up with the functional and quality requirements. This went smoothly. We agreed that the new model must beat the legacy one on the language currently supported and reach at least 50% accuracy on multilingual documents (European languages only).
Speed came up as well: ideally, the model should match the latency of the current system (below a few seconds). We kept an option open, though – if a longer processing time noticeably improves accuracy, it’s worth considering.
10. Question list
We wrapped up the session by going through the list of questions I had prepared beforehand. The client had done his homework, so we could finalize the key assumptions and agree on them. Most topics had already come up during the workshop – this was a last pass to make sure we were aligned. The questions dealt with data volume and format, quality expectations and who owns which tasks.
11. Wrap-up
To close the session, I summed up what we had accomplished, laid out the next steps and explained how long we would need to process everything we had learned and finalize the offer.
From then on, the client’s part was finished and ours was only starting. With the right inputs in hand - requirements, constraints and a solid grasp of both the process and the platform - we could begin preparing a genuinely tailored proposal.
Workshop outcomes: more than just a concept
The workshop let us collect a huge amount of crucial information in a short time - all of it necessary for preparing a precise proposal and project report.
Multilingual support
We discovered several key project requirements nobody had mentioned before, such as the need for a multilingual solution. It might have surfaced through the pre-workshop questionnaire, sure; but what if that question had been left off the list? We would have run into an unpleasant surprise during implementation.
Feedback mechanism
We also found out that the platform already has a customer feedback mechanism for checking and correcting model predictions. In our preliminary project scope offer, this was one of the milestone components. Knowing this, we could safely drop that milestone, which considerably reduced the implementation effort and the overall project cost.
Data quality
On top of that, we learned how the system’s data is used within the client’s business. This confirmed our assumption that data validated by users is of exceptionally high quality. Users simply can’t afford to be careless with it, since so much depends on it – the accuracy of the data directly affects what they deliver.
Choosing the right approach to Key Information Extraction
In the workshop, we looked at various ways of automating key information extraction from invoices – among them SaaS platforms, commercial LLMs and custom models trained on the client’s data. This topic is strategically essential and technically nuanced, so it calls for a separate, detailed comparison of each approach in terms of cost, risk, requirements and business fit.
Final offer: What the client received?
1. Education
Most importantly, we walked the client through planning an ML project so that he would clearly see how such a project works, how it differs from typical IT projects and which key elements require particular attention.
2. Idea validation
In the workshop, we examined the data that represents the core problem and learned how the target system should work, how much training data is available and what other limitations exist. Based on that, we could judge whether machine learning could solve the problem and how to tackle it to lower the risk of failure and maximize the client’s long-term savings.
3. Cost analysis
The delivered report contained an in-depth analysis of several solution variants. One of its main results is a clear picture of the development effort and long-term operating costs of the different solution options.
Using this data, the client could estimate the total cost of ownership (TCO) and weigh it against the expected business impact – cost savings, more efficient processes or revenue growth. This gives him a firm basis for calculating the investment’s ROI and forecasting when it will break even. Rather than deciding based on gut feeling or vendor promises, the client works with concrete figures tailored to your specific context.
4. Project plan
The final offer consisted of five project phases.
- Phase 1: Analysis of the data
- Phase 2: Annotation of training data
- Phase 3: The main PoC phase
- Phase 4: Deployment to production
- Phase 5: Automating retraining & monitoring workflows

Every phase was described in detail, together with pricing and duration estimates. This gave the client a complete view of the total costs, the estimated delivery timelines and the option of speeding things up by increasing the team’s involvement.
Acceptance criteria
For every project phase, prerequisites were clearly defined – what has to be prepared or delivered before the phase can start. We outlined the implementation path, key milestones and acceptance criteria of each phase. Responsibilities and points of contact were specified for both sides. The documentation also contained time estimates and the expected number of engineer-days needed for every stage.
Time estimates
The offer included a detailed technical plan spanning the whole project lifecycle. The full plan was laid out on a timeline and visualized to show the order of tasks and which phases could run in parallel.
This was more than a proposal. It was a project roadmap ready for deployment.
A workshop is not a cost; it’s an investment
Many companies thinking about machine learning may see a workshop as an “optional” step that can be skipped in favour of going straight to development. Yet a well-planned and properly run workshop is among the most important factors that directly affect whether a project succeeds.
Rather than kicking off the project on a hunch, the client got a real, informed choice covering operating costs, maintenance complexity, retraining options and an estimated ROI. This removes needless risk and lets data, not guesswork, drive business decisions.
For us, the workshop is also where trust begins – not through promises of “AI magic,” but through a transparent, pragmatic approach grounded in experience, expertise and solving problems together.
If you’re wondering whether an artificial intelligence project makes sense for your organization, a workshop is the best way to find out. At worst, you leave with a clear map of the risks and an understanding of what you don’t know yet. At best, you start a well-structured, realistic and financially sound project from the very first day.
Summary
What does choosing the workshop give you?
- Identify real system constraints early on, ruling out particular off-the-shelf tools right away (e.g., because they lack multilingual support)
- Recognize the hidden value of existing data and validation mechanisms, which let us set up an effective feedback loop for the ML model - something the client didn’t at first see as a strategic advantage
- Uncover subtle but important requirements, such as mapping model predictions onto document layouts or being able to expand to new languages and countries - essential for UX and future growth
- Build shared understanding of the problem, leading to client expectations and better readiness to make well-informed investment decisions
- Accelerate estimation and execution – within a few days, the client had a detailed project plan, a cost breakdown and a recommended solution
- Verify the idea before implementation starts, which increases your project’s chances of success
- Verify that the implementation cost and operating costs stay within budget
- Get cost estimates to calculate ROI
All of this happens before you spend a fortune on a project that hasn’t been properly thought through.
We’d be glad to guide you through the discovery process, evaluate feasibility and help you make informed, data-driven decisions about your next ML initiative.