Capturing data about manual processes and manual machining steps is important in manufacturing for better traceability, optimization, and better planning. Current manufacturing research focuses on sensor-based recognition of manual activities across multiple tools or power tools, but little on recognition within a versatile power tool type. Due to the strong influence of operator skill on process performance and consistency as well as many disturbance variables, activity recognition is a challenge in manual grinding. It is unclear how accurately manual activities can be recognized within one handheld grinder type across diverse trials. Therefore, this article investigates how manual activities can be recognized in diverse trials within an angle grinder type in a leave-one-trial-out cross-validation in comparison to classical cross-validation to identify the effect of diverse trials with four different classifies. An experimental study was conducted to collect measurement data with data loggers attached to two angle grinders, four manual activities with different abrasive tools, and three operators. Results show very good accuracies (97.68%) with cross-validation and worse accuracies (70.48%) with leave-one-trial-out cross-validation for the ensemble learning classifier. This means that recognition of the four chosen manual activities within an angle grinder is feasible but depends on how much the trial deviates from the reference training data. For further research on activity recognition in manual manufacturing, we propose the explicit consideration and evaluation of disturbance variables and diversity in data collection for the training of machine learning models.