When milling, which is machining in the mill, the cutting bit size determines the rpm. When turning down bar stock in the lathe, the diameter of the bar stock determines the speed of the machine.
Other factors to consider when determining the speed and feed rate:
The material will determine what is known as Surface Feet Per Minute, (SFPM). This is a key unit when calculating the rpm of a machine.
RPM = (3.82 X SFPM) / Diameter
Calculations get you close, not exact!
If you are drilling in the lathe, and the drill bit is stationary, the drill bit diameter determines the speed, not the bar stock diameter.
Typically chatter is caused by too fast an rpm or too slow of a feed rate, so speed up the feed rate or slow down the rpm.