Set, circle, coterie, clique can all denote a more or less carefully selected or exclusive group of persons having a common interest.
Set applies to a comparatively large group, especially of society men and women bound together by common tastes.
Circle implies a common center of the group (as a person or a cause that draws persons to him or it) or a common interest, activity, or occupation.
Coterie stresses the notion of selectness or of congeniality within the small circle; clique heightens the implication of an often selfish or arrogant exclusiveness.