Matter, substance, material, stuff are comparable when they mean what goes into the makeup or forms the being of a thing whether physical or not.
In the relevant sense matter basically denotes that of which all physical objects are made, but to the physical scientist this may imply the component of the observable universe that includes among its properties extension, inertia, and gravitation and is held to consist ultimately of relatively few kinds of elementary particles, to be convertible into energy, and together with energy to form the basis of objective phenomena while to the philosopher it tends to imply an unordered material substratum distinguishable on the one hand from immaterial qualities (as spirit or energy) and on the other from form or formed bodies.
In more general use matter often specifically applies to a particular portion or kind of physical matter.
Substance usually implies a particular kind of matter and often one of known chemical or physical nature or it may distinctively suggest the particular matter or kind of matter that enters into the composition of something or gives it its characteristic properties.
Material applies basically to matter or substance as a constituent of physical and especially of made things.
But often it subordinates physical nature to the fact of being made and then may imply the idea of actuality or a basis in actuality rather than physical substance; thus, raw materials are usually physical substances capable of refinement or manufacture but they also may be events, ideas, or facts capable of further use (as in literary creation).
Stuff may replace material or substance in reference to constituent materials or substances, but more often it refers to all the parts, parcels, objects, or items that make up an aggregate or a whole; in both cases it is likely to imply indeterminateness and suggest vaguely if at all the nature of the constituent materials or parts.
Sometimes, distinctively, stuff carries an inherent implication of inferiority.