The first 20% of a putt is not rolling at all

The ball you have just struck is not actually rolling. It hops, it skids, and only then does it start to roll.

The hop

A putter has 2–4° of loft, so the ball lifts very slightly off the surface. It is projectile motion, so the distance is:

d = v₀² sin(2β) / g

At a launch angle of 2° and an initial speed of 3 m/s that is about 6 cm. We are talking about centimetres — but while the ball is in the air there is no contact force, so seen from above it travels perfectly straight.

The skid

On landing the ball has almost no spin. The contact point slides, and sliding friction slows the ball down while spinning it up at the same time. The slip disappears and pure rolling begins at the moment the speed reaches:

v_roll / v_land = 1 − (2/7)(1 − r)

where r is the amount of topspin at launch. With no spin at all that is 5/7. This ratio is an exact result for a rigid sphere and does not depend on the sliding friction coefficient.

What the friction coefficient does determine is the length of the skid, and for typical values that comes out at 15–20% of the whole putt.

Which is why a putt breaks late

Sliding friction also resists sideways slip, so the break is suppressed during this phase. On top of that the ball is moving fast, which keeps the curvature of the path low in its own right.

For a 6 m putt on a 2° side slope, the sideways deviation at 25% of the way was 5.6 cm in a model that ignores the skid, and 4.1 cm in one that includes it.

The feeling that a putt "goes straight at first and breaks hard at the end" is backed up by the physics exactly as you would expect.