“Our supplier delivers fine, just late.” In most procurement meetings, that sentence is the only supplier evaluation available. The buyer, the warehouse operator and the planner all share it, and nobody can put a number on it. Yet your ERP already holds the purchase orders, the goods receipts, the quality inspections and the invoices. What is missing is a method for turning those rows into a scorecard you can put in front of a supplier without being challenged on every figure.
This guide is written for procurement, supply chain and IT leaders in mid-sized manufacturers and distributors. It sets out six KPIs, their formulas, the ERP fields to query, a weighting scheme with a worked (fictional) example, and a governance loop. It complements our article on supplier risk scoring in the ERP: where that article maps exposure (financial, geographic, regulatory), the scorecard measures the actual operational performance delivered month after month.
Why suppliers are rarely rated with ERP data
Three reasons explain why the scorecard is often a hand-fed Excel file, or no scorecard at all.
The data is scattered. The order lives in the purchasing module, the receipt in inventory management, the inspection in the quality module (or a third-party tool), the invoice in accounts payable. No standard screen reconciles them by supplier and by order line.
Dates are not historized. This is the sneakiest problem. The supplier confirms a date, then pushes it back twice by phone. The buyer edits the order line date so the production schedule stays consistent. The original date is overwritten, and the ERP can no longer say whether a delivery was late against what you asked for or against the last promise.
Nobody owns the topic. Procurement thinks “price”, logistics thinks “receipt”, quality thinks “non-conformances”. The scorecard sits at the intersection of the three, so it belongs to none of them.
The fix is organizational before it is technical: decide which dates are kept, who enters what, and which KPI is the reference. The rest is just queries.
The 6 core KPIs and their formulas
Six KPIs are enough to start. Beyond that, you drown the supplier (and your own teams) in numbers nobody manages against.
OTIF (On Time In Full)
OTIF measures the share of order lines delivered on time and complete.
OTIF = lines delivered within the window AND within the quantity tolerance / lines due in the period
Three decisions to lock down before calculating anything:
- The tolerance window. Delivery on the exact day, ± 1 day, ± 3 days? It depends on the type of purchase. A just-in-time component cannot carry the tolerance of a maintenance consumable.
- The quantity tolerance. Is a delivery at 98% of the ordered quantity “complete”? Many companies accept a range (for example -2% / +5%) aligned with the tolerances already configured for invoice matching; see our article on three-way match and its tolerances.
- The counting unit. Order line or whole order? Counting by line is finer and fairer. Counting by order heavily penalizes a supplier that delivers 19 lines out of 20 on time.
The major trap is the choice of reference date. An OTIF calculated against the date confirmed by the supplier is flattering, since it measures their ability to keep their own promise. An OTIF calculated against the date requested by your planning measures what your plant actually needs. Keep both, name them differently (“confirmed OTIF” and “requested OTIF”) and watch the gap: if it is wide, the supplier is confirming dates that do not suit you and you cannot see it.
Another trap is the partial receipt. If the supplier delivers 60% on time and the balance three days later, the line is not “in full” on the due date. Decide once and for all whether the line counts as a failure or the balance is assessed separately, and document the rule.
Last trap: how the number is read. An OTIF of 92% that counts deliveries within ± 3 days does not say the same thing as 92% on the exact day. Without the definition next to it, the percentage is meaningless, and comparisons between suppliers or sites become misleading.
Actual lead time vs contractual (or ERP) lead time
OTIF is binary: on time or not. Lead-time variance tells you by how much.
Lead-time variance = actual receipt date - reference date (in calendar or working days, your choice, but consistent)
Two sub-indicators are useful:
- The average observed lead time per item or family, compared with the replenishment lead time set in the item master. If it is systematically longer, your net requirements are computed on a wrong assumption and you create stockouts or excess safety stock.
- The dispersion (standard deviation, or simply the longest lead time observed over 12 months). A supplier at 10 days “on average” who swings between 4 and 25 days costs more in safety stock than a stable supplier at 12 days.
Feeding information back into the ERP creates real value here: the observed lead time can update the planned lead time, which improves requirements planning.
PPM and incoming non-conformance rate
PPM (parts per million) relates the number of defective parts to the number of parts delivered, on a per-million scale.
PPM = parts rejected or non-conforming / parts received × 1,000,000
It is relevant for high-volume parts and components. For low-unit-volume purchases (machines, services, material in lots), prefer the non-conforming lot rate:
NC lot rate = lots rejected or accepted under deviation / lots received
Points to watch:
- Distinguish defects found at receipt from defects found later in production or at the customer. The first sits in the quality inspection module; the second often comes back through another channel. The basic scorecard only measures the first, and you should say so.
- Count deviations: a lot accepted under deviation is not a conforming lot.
- Never compare PPM calculated on different inspection plans (100% inspection versus sampling). Sampling lets through defects that exhaustive inspection catches.
Invoice vs order price variance
Price variance = (invoiced unit price - ordered unit price) / ordered unit price, weighted by quantity.
Your three-way match already computes part of this indicator: every out-of-tolerance variance blocks the invoice. You only need to aggregate it by supplier. Two pieces of information matter:
- The frequency of invoices with a price variance (for example, 1 invoice in 8 blocked on price).
- The direction: a supplier that systematically invoices above the order, without an amendment, tells you something different from one that forgets a discount.
Do not mix this with contractual price revision (indexation): that is planned and must not lower the score.
Responsiveness
Responsiveness measures how quickly the supplier answers requests: RFQs, order confirmation requests, technical questions.
Responsiveness = average time between sending the request and receiving a usable answer
This is the hardest KPI to automate, because answers often arrive by email. It becomes reliable when exchanges go through a supplier portal or an ERP sourcing module that timestamps both sending and reply. If you have no such channel, measure a single simple event: the time between sending the purchase order and receiving the acknowledgement. To go further on portals, see our article on digitalizing procurement with a vendor portal.
Dispute and credit note rate
Dispute rate = invoices leading to a dispute or credit note / invoices received
If you can, add the credit note amount relative to the invoiced amount. Dispute causes (quantity, price, quality, duplicate, payment terms) are more useful than the overall rate: they point to the root of the problem and feed the action plan. It is also a hidden-workload indicator: every dispute consumes hours of accounts payable and procurement time.
Where to find the data in your ERP
The table below is deliberately generic. Exact object, table and report names depend on your version, your configuration and the extensions installed: validate them with your integrator.
| Data to capture | SAP (S/4HANA) | Dynamics 365 (Finance & SCM / Business Central) | Odoo |
|---|---|---|---|
| Requested and confirmed date | Purchase order, schedule lines, supplier confirmations | Purchase order line, requested and confirmed dates | Purchase order line, expected date, confirmation date |
| Actual date and quantity | Goods receipt against purchase order | Product receipt | Receipt (incoming picking), quantity delivered |
| Non-conformances | Quality inspection at receipt | Quality management, quality orders | Quality checks (Quality module) |
| Invoiced price | Invoice verification, matching | Vendor invoice, matching | Vendor bill, comparison with purchase order |
| Disputes and credit notes | Blocked invoices, credit memos | Invoices with variances, credit notes | Vendor credit notes |
| Responsiveness | Requests for quotation (purchasing) | Requests for quotation | Calls for tenders, requests for quotation |
Two field notes.
Check date historization first. On your ERP, is the order line date overwritten on every change? Is there a change history for orders? If not, you cannot compute “requested” OTIF retroactively. In that case, start capturing now (a dedicated “initial requested date” field, frozen when the line is created), and accept starting the scorecard with three months of clean data rather than twelve months of doubtful ones.
Joining by supplier and by line is the real difficulty. Most of the time, a small data warehouse or a BI report brings these five sources together. The effort is a few days for a first scope, not several months, if you stick to the six KPIs. Our article on supplier relationship management (SRM) in the ERP details what specialist solutions add when the scope widens.
Weighting and computing an overall score
The method in three steps
- Normalize each KPI on a 0 to 100 scale, with an explicit target (100) and floor (0).
- Weight according to the criticality of the purchase.
- Sum to get the overall score, and classify as green, amber or red.
Worked example (fictional, for teaching purposes)
Take a fictional supplier of machined components and one quarter of imaginary data. The figures below illustrate no real supplier and are not market benchmarks.
| KPI | Result | Score /100 | Weight | Contribution |
|---|---|---|---|---|
| Requested OTIF (± 1 day) | 88% | 80 | 30% | 24.0 |
| Average lead-time variance | +2.5 days | 65 | 15% | 9.8 |
| Receipt PPM | 1,800 | 70 | 25% | 17.5 |
| Invoice price variance | 1.0% of invoices | 90 | 10% | 9.0 |
| Responsiveness | 3 days | 60 | 10% | 6.0 |
| Disputes and credit notes | 4% of invoices | 75 | 10% | 7.5 |
| Overall score | 100% | 73.8 |
The rule for converting a result into a score is yours to set and publish. For example, for OTIF: 100 for a result at or above target, 0 for a result at or below the floor, linear interpolation in between. What matters is not the formula you pick, but that it is written down and stable from one quarter to the next.
Green, amber, red thresholds
A common set of thresholds, to adapt:
- Green: 80 and above. High-performing supplier, routine relationship.
- Amber: 60 to 79. Targeted action plan on the weakest KPI.
- Red: below 60, or a knock-out criterion (for example a critical non-conforming lot). Formal improvement plan, and a path toward delisting if there is no improvement.
Our fictional supplier, at 73.8, is amber: the action targets responsiveness and lead-time variance, not price.
Adapt the weights to criticality
A single set of weights for all purchases is a classic mistake. Adjust by category:
| Purchase category | KPIs to overweight |
|---|---|
| Strategic material or component, single source | OTIF, PPM, actual lead time |
| Consumables, supplies | Price, disputes, responsiveness |
| Services | Responsiveness, disputes, milestone compliance |
| Resale goods (distribution) | OTIF, service level, price variance |
Three or four weighting profiles are enough. Beyond that, the system becomes unreadable.
A score must be explainable
A supplier who does not understand their score will contest it, rightly. Three simple practices: share the definition of each KPI and the scoring rule before the first publication, provide line-by-line detail in case of dispute (the order, the date, the quantity), and never change a rule mid-quarter. A transparent scorecard is a dialogue tool. An opaque one is a tool for distrust.
The governance loop: review, action plan, decision
The calculation is only a third of the job. A scorecard with no decision loop is just one more dashboard.
Every month, procurement checks the alerts: any supplier that turns red or loses more than 10 points in a month (indicative threshold) triggers a conversation.
Every quarter, a procurement, logistics and quality review examines:
- critical suppliers (high volume or single source), systematically;
- the 5 lowest scores and the 5 biggest improvements;
- the status of ongoing action plans.
For each amber or red supplier, a three-line action plan: root cause identified with the supplier, action and owner, deadline and tracking KPI.
The link to structural decisions is what gives the scorecard weight:
- Approval: a minimum score over two quarters conditions entry into or retention on the approved supplier list.
- Volume allocation: for an equivalent offer, marginal volumes go to the best scores.
- Renegotiation: the scorecard brings facts (observed lead time, PPM, disputes) to open a discussion on penalties, consignment stock or price. A green supplier is negotiated with as a partner; a red supplier is negotiated with on quantified commitments.
The quarterly review is also the simplest building block of evidence for a documented supplier due diligence process; see our article on supplier risk scoring.
Common mistakes and countermeasures
1. Measuring OTIF against the confirmed date only. Countermeasure: freeze the requested date when the line is created, compute both OTIFs and track the gap.
2. Not documenting the tolerance window. Countermeasure: write the definition next to each percentage (“OTIF at ± 1 day, quantity tolerance -2% / +5%”) and share it with the supplier.
3. Applying the same weights to all purchases. Countermeasure: 3 or 4 weighting profiles by criticality, validated by the head of procurement.
4. Publishing a score the supplier cannot verify. Countermeasure: line-level detail access, published scoring rules, no changes mid-quarter.
5. Calculating without acting. Countermeasure: dated review calendar, an action plan for every amber or red supplier, an explicit link to approval and renegotiation. If the score never has consequences, teams stop entering data properly and the scorecard degrades.
30-day implementation checklist
Week 1, scoping
- Choose the scope: your 20 to 30 most important suppliers, not the whole panel.
- Fix the definitions: tolerance window, quantity tolerance, counting unit (line or order).
- Name a scorecard owner.
Week 2, data
- Check date historization in the ERP, create the “initial requested date” field if needed.
- Map where receipts, quality, invoiced price and disputes live.
- Extract twelve months (or as much clean data as available) for a pilot supplier.
Week 3, calculation
- Compute the six KPIs for the pilot supplier and check a few lines by hand.
- Define the scoring rules, the weights per category and the thresholds.
- Extend to the other suppliers in scope.
Week 4, sharing and governance
- Present the method to the suppliers in scope before publishing scores.
- Hold the first review and decide the first action plans.
- Schedule the monthly and quarterly reviews in calendars.
After 30 days you do not have a perfect system: you have an explainable scorecard that runs, and a decision loop. Refinements (extra KPIs, automation, a supplier portal) come next, based on what the reviews teach you.
To structure your whole selection and monitoring approach, download our ERP evaluation grid: 30 criteria over 100 points to compare three vendors side by side. For the procurement side, also read our guide to supplier relationship management (SRM) in the ERP.
