Twist insertion in soft filaments is an efficient way to produce the twisted and coiled polymer (TCP) muscles. The study on the formation process of highly twisted filaments is significant for understanding the mechanics of filament-based artificial muscles. A novel in situ torsion tester is developed for measuring the torsional behavior of monofilaments under an array of axial forces. The torque transducer consists of a flexural pivot attached by a mirror, the rotation angle of which is measured by an electronic autocollimator. The calibration results demonstrate perfect linearity between the torque and the angular displacement. The torsion tester combines a vast torque capacity of about 8.59 × 10−3 Nm with a resolution of about 1.02 × 10−8 Nm. Torsion experiments are performed on nylon 6 monofilaments under various axial forces. The critical torques for the first and the secondary instability of monofilaments under different axial forces are obtained. Three stability criteria of filaments subjected to torsion and tension, due to Timoshenko, to Ross, and to Dwivedi and co-workers, are accessed within the context of measurement data. The critical torque for the first instability predicted by the Timoshenko model agrees with the experimental results. The critical torques for the secondary instability can be well predicted by both the Ross and the Dwivedi et al.'s models, although the latter model gives a more reliable prediction.