Why anything is milled at all
Small particles extract faster, which is what a three-minute teabag needs. That is a real engineering reason rather than a swindle: bagged material is cut small because the format demands it, and a bag full of whole chamomile heads would brew weakly.
What milling costs
Two things. Surface area rises, so aroma oxidises away faster in storage and bitterness extracts faster in the cup. And the structure that let you assess the material is destroyed: whole chamomile heads are recognisable and their condition is visible, while chamomile dust could be anything, dried at any temperature, in any year.
Where it matters most
Flowers, by a wide margin. Chamomile, rose, chrysanthemum, honeysuckle and lavender are all assessed by looking at them, and all lose that once cut. Aromatic leaves are the next most affected. This is the group where paying for whole material buys something concrete.
Where it matters least
Roots, barks and roasted grains are cut or roasted whatever you buy, and nobody assesses a chicory root by eye. Buying these loose gains you dosing control and very little else, and the honest advice is that the format argument is weaker here.
The middle option
An empty fill-your-own filter bag is the compromise that gets most of both: your material, chosen and assessed, in a format that behaves like a bag. It costs almost nothing and is under-used.