Precision Is a Variance Problem, Not an Accuracy Problem

A part that comes out consistently 0.002 in oversize is an easy problem: change the offset and it is solved forever. A part that comes out somewhere between 0.001 in under and 0.002 in over, with no pattern, cannot be fixed by any offset. Holding tolerance is mostly about finding and removing that scatter.

Accuracy and Repeatability Are Different Problems

Accuracy is how close you land to the target. Repeatability is how tightly your results cluster, wherever that cluster happens to sit. They call for opposite responses.

• Consistent bias. Every part is off by roughly the same amount in the same direction. Adjust the offset, verify, and move on. This is not a quality problem.
• Scatter. Parts land on both sides of nominal with no repeating pattern. No offset helps. Something in the process is changing between parts, and until you find it, every tight tolerance is a coin flip.

When a job starts producing occasional bad parts, the first question is which of the two you are looking at. Measure ten consecutive parts and write the numbers down. The shape of that list tells you what to chase.

Where the Variation Actually Comes From

Four sources account for most of it, and they behave differently enough that they are usually distinguishable from the data.

• Tool wear. Drifts in one direction, slowly, and accelerates near the end of tool life.
• Thermal growth. Drifts in one direction across the first hour of a shift, then stabilizes.
• Deflection. Varies with how much material the tool is taking, so it changes with stock condition rather than with time.
• Fixture and setup repeatability. Random from part to part, which is exactly what makes it the hardest to catch and the most damaging.

Thermal Growth: The One Nobody Budgets For

Metals expand predictably, and at the tolerances a precision job works to, the numbers are not small. Steel expands roughly 6.5 millionths of an inch per inch per degree Fahrenheit. Aluminum is roughly twice that.

Work it out for a real part. A 10 in aluminum component that warms 20°F during machining grows about 0.0026 in. If the print calls for ±0.001 in, the part was out of tolerance before the tool touched it a second time, and the measurement you take at the machine, on a warm part, will not match the one the customer takes at 68°F.

• Warm the machine up before running precision work; a cold machine is a moving machine
• Keep coolant temperature stable, since coolant is what sets the part temperature
• Let parts normalize before final inspection, or correct the reading for temperature
• Take a shop that runs tight tolerances seriously about ambient temperature; a door that opens onto a loading dock in winter is a measurable process variable

Tool Wear Shows Up as Dimensional Drift

As an end mill wears, its effective cutting diameter gets smaller. Under cutter compensation, that means pockets come out progressively undersize and bosses progressively oversize, in a slow one-way drift that looks like the machine is losing calibration.

The fix is not a smarter offset strategy. It is retiring the tool before the drift reaches the tolerance band, which means knowing how many parts a tool is good for and treating that number as a rule rather than a guideline. On a job with meaningful tolerance, the tool should be changed on a count, not when someone notices the finish getting worse.

Runout and Deflection

Two mechanical effects sit underneath everything above.

Runout means the flutes are not all on the same circle, so one of them cuts deeper than the rest. The part sees an effective diameter larger than the tool actually is, the loaded flute wears faster than the others, and the dimension moves as it does. Runout is worth measuring at the tool tip rather than at the holder, because a small angular error at the spindle becomes a large one at the end of a long tool.

Deflection means the tool bends away from the cut under load, leaving material behind. It is worst on long, slender tools and heavy radial cuts, and it is why a wall machined in one pass is rarely straight. The conventional answer is a spring pass: a final light cut at the same commanded position, which removes the material the deflected tool left. It works because the load on the spring pass is small enough that the deflection is small. Finishing end mills for that last pass are in Corner Radius and High-Performance End Mills.

Surface Finish, With the Units Straight

Surface roughness is quoted as Ra, and the single most common error in shop conversation is mixing the two unit systems. They are not interchangeable and they differ by a factor of about forty.

• Ra 125 µin (3.2 µm) is a general machined finish
• Ra 63 µin (1.6 µm) is a good finish milling or turning result
• Ra 32 µin (0.8 µm) is achievable with a sharp tool, a rigid setup, and a light finish pass
• Below that generally belongs to grinding, honing, or lapping rather than to a milling cutter

Finish is driven primarily by feed per revolution and by nose radius in turning, and by feed per tooth and by the condition of the edge in milling. A worn edge does not produce a fine finish at any speed. Our TechTalk on why surface finish matters goes into what those numbers mean for the part in service. Insert nose radius options across all three lines are in Turning Inserts.

Inspect Where It Prevents Scrap

Inspection after the run tells you how many parts you lost. Inspection during the run stops you losing them.

• First article, fully. Every dimension on the print, before the second part is cut.
• Sampling on a schedule through a long run, on the features that are actually tight rather than on all of them.
• Check after every tool change, because that is where steps in the data come from.
• Trust the gauge before you trust the reading. If two people measuring the same feature disagree by a meaningful fraction of the tolerance, the measurement system is part of your scatter and needs fixing before anything else does.

Consistency Is a Procedure, Not a Skill

The most common source of part-to-part scatter in an otherwise sound process is the setup being done slightly differently each time. Clamping sequence, torque on the clamps, how chips are cleared from the locating surfaces, which edge is indicated against. Write the setup down. A one-page sheet with the fixture, the clamp order, the tool list, the offsets, and the parameters turns a skilled operator's good habits into something the whole shop repeats.

What to Take Away

Before adjusting anything, measure ten parts and look at the shape of the numbers. A bias is an offset. A slope is heat or wear. Noise is the setup. Retire tools on a count rather than on appearance. Take the temperature of the part seriously. Get the units right when you talk about finish. And write the setup down, because consistency that lives only in one person's hands is not a process.

Keep Learning

Continue with these related TechTalk deep-dives from the FM Carbide engineering team:

• Why Surface Finish Matters in Engineering
• Carbide End Mill Feeds and Speeds Explained

Need help choosing the right tool? Browse our carbide tooling catalog or talk to our engineering team — FM Carbide helps you machine better!