Vibecheck Phase 2

Status Quo

Motius successfully completed a Proof of Concept (as of August 2025) of the Vibecheck acoustic end-of-line testing.

• Structure-borne sound microphone: Capture of sound signals during ~20s dynamic test profile (two directions & two speeds, 4-5s each)
• ML algorithm: Optimized classification through extended datasets and augmentation
• Test sequence: Dynamic profile with forward/backward rotation at various RPM, integrated into LabVIEW & TestStand
• Simple User Interface: Workers can confirm / correct results, with a simple green/red display in LabVIEW
• Data processing: .wav (10s samples) → Log-Mel spectrograms (up to 8kHz) → CNN/Autoencoder
• K series gearbox: PoC was validated on K series helical-bevel gear motors

Based on 650+ experiments, as of August 2025, we achieved a reliable OK/Not OK classification with >90% accuracy (Confusion Matrix validated).

Defect Types

• Input: .wav recordings from 20s dynamic motor test sequences
• Preprocessing: 5s sample extraction with overlap
• Feature Extraction: Log-Mel spectrograms (max frequency: 8kHz)
• Normalization: Per-sample spectrogram normalization
• Time-Frequency Representation: Horizontal time axis, vertical frequency axis

Dataset Characteristics

The comparatively low percentage of valid experiments points to a need for automation.

The model training required a balanced dataset, which means that roughly the same amount of defect samples should be used.

Performance Metrics on Test Set

Performance metrics are only calculated for valid experiments:

• Sensitivity (Recall): ~89-94% - Critical for catching defects
• Specificity: ~81-100% - Important to minimize false alarms
• F1-Score: ~84-97% - Balanced performance indicator

Current IT Integration

• Connectivity & Deployment: Direct network connection to backend running in Motius AWS infrastructure
• Test bench integration: Custom Python module is called in the test process
• UI: Labelling interface, display of results with green/red status, allow marking experiments as irrelevant, and highlight important information (such as spectrograms that lead to not-OK classifications)
• Workflow:
  • 20s total data recording time, ~30s in total (including processing times)
  • Additional reminder pop-up for attaching the body sound sensor reduced the number of invalid experiments from 25% to 5%
• Laser vibrometer: Comparative studies show equivalent results to structure-borne sound microphone
• Analysis App: VibeCheck Web App for manually analyzing, re-labelling datapoints, and marking experiments as invalid (thereby excluding them from training)

Laser Vibrometer

We benchmarked a Polytec laser vibrometer with auto-focus against the structure-borne sound microphone. The vibrometer provided similar sensor readings, without contact to the motor.

Phase 2

Extending VibeCheck Algorithm

The successfully validated PoC algorithm currently supports one gearbox type (bevel gear motors). In Phase 2, we want to support a larger variety of products, and roll out to multiple quality cells:

• Extended datasets: Further training with additional experiments for bevel gear transmissions
• Data augmentation: Audio augmentation (Pitch, Time Stretch) + spectrogram augmentation (Zoom, Brightness, Mixup, Erasing)
  • The augmentation can improve the model generalization capability and can also enforce that the model learns features instead of memorizing training data features
  • Data augmentation will be especially helpful for training new model versions for new gearbox types, which initially only have a small dataset
• Continuous learning: Automatic model updates based on new production data
• Performance monitoring: Continuous monitoring of classification accuracy
  • Explainable AI: Feature importance and decision visualization for experts
  • Historic data: Show history of experiments and model performance
• Invalid experiment detection: Additional script or model for detecting faulty experiments
  • RMS Threshold: Detect microphone disconnect (sound intensity too low)
  • Spectral Analysis: Identify unusual frequency patterns
  • Future Enhancement: Dedicated ML model for experiment validation
• Advanced Architectures: Research transformer models for sequence modeling

Data Augmentation Strategy

Model Architecture

Before scaling to more cells, the team needs to decide whether to extend a single model or train multiple gearbox-specific models.

The team will likely train multiple models and test their performance compared to a single, bigger model.

Additionally, the architecture of the models could be adapted, after the first tests:

Model Training Infrastructure

Even if the team decides to only train one model, a production deployment requires more infrastructure than the current PoC.

MLflow is an open-source platform for managing machine learning models. It can tie into existing PoC infrastructure (storage, database) and add model versioning, training with new data, and monitoring performance.

graph TB
    classDef primary fill:#64CEE4,stroke:#64CEE4,stroke-width:2px,rx:10px
    classDef default fill:none,stroke:#64CEE4,stroke-width:2px,rx:10px
    classDef defaultBackground fill:#FFFFFF44,stroke:none,rx:20px
    classDef primaryBackground fill:#23BAD933,stroke:none,rx:20px

    subgraph Gearbox["Quality Cell"]
        GB1[K-Series Gearboxmotor]
        GB2[S-Series Gearboxmotor]
        GB3[Servomotor]
        AUDIO[Sensor]
        TS[TestStand Python Node]
    end

    subgraph Web["Web Services"]
        ADMIN[VibeCheck Admin Interface]
        REST[VibeCheck REST API]
        UI[MLflow Admin Interface]
    end

    subgraph MLflow["MLflow Tracking Server"]
        MLT[MLflow Tracking Server]
        MLB[MLflow Backend]

        subgraph Model["Model Registry"]
            MR[MLflow Model Registry]
            V1[Model v1.0<br/>K-Series]
            V2[Model v2.0<br/>K-Series]
            V3[Model v1.0<br/>S-Series]
        end
    end

    subgraph "Storage"
        S3[MinIO<br />S3-compaible storage]
        DB[Database]
        ARTIFACTS[Model Artifacts<br/>- Trained Models<br/>- Feature Extractors<br/>- Preprocessors]
        LOGS[Training Logs<br/>- Metrics<br/>- Parameters<br/>- Audio Samples]
    end

    Gearbox:::primaryBackground
    Web:::primaryBackground
    MLflow:::primaryBackground
    Storage:::primaryBackground
    Model:::defaultBackground

    %% Connections
    TS <-->|API Request<br/>Sensor Data| REST
    REST <-->|API Request<br/>Model Inference| MLT
    MLT -->|Fetch Model| MR
    MR --->|Load Artifacts| S3
    S3 -->|Model Files| MLT
    MLB --->|SQL| DB
    ADMIN --->|SQL| DB

    %% Model versions
    MR --> V1
    MR --> V2
    MR --> V3

    %% Storage connections
    MLT --->|Store Artifacts| S3
    S3 --> ARTIFACTS
    S3 --> LOGS

    %% UI connections
    UI -->|View Models<br/>Compare Versions| MR
    UI -->|View Metrics| MLT
    UI --> MLB

    %% Gearbox data flow
    GB1 --> AUDIO
    GB2 --> AUDIO
    GB3 --> AUDIO
    AUDIO -->|Input Features| TS

    class GB1,AUDIO,TS,ADMIN,REST,S3,DB,ARTIFACTS,V1 primary

Highlighted components in blue are already in place from the PoC, the other components will be added in Phase 2.

Deploying to SEW IT Infrastructure

Next, the algorithm needs to be deployed in SEW's IT infrastructure:

• VM vs Cloud: Decide with an expert from SEW's IT whether we deploy to Azure or into a VM
• Database & Storage Migration: Database and *.wav file storage needs to move to either Azure or a VM
• Model Versioning: Traceable versioning with rollback functionality

For the deployment, SEW needs to provide infrastructure with these parameters:

Test Strategy

The deployed algorithm then needs to be tested on new product types and in new quality cells:

• Iterative Testing: Multiple test cycles with SEW experts
• Model validation: Confusion Matrix validation against SEW expert classifications
• False Negative Prevention: Testing & model validation need to ensure that false negatives are very unlikely
• False Positive Minimization: Too many false positives lead to additional manual work
• Performance Benchmarks: At least 90% classification accuracy
• Integration Testing: Complete LabVIEW pipeline validation

To structure this testing, the team will create an updated test strategy for a production-ready rollout.

Rollout to new Quality Cells and Product Types

An improved ML algorithm, hosted on SEW infrastructure, after proper testing enables SEW to roll out VibeCheck on their own:

1. A process owner at SEW installs the required sensor and TestStand software in a new quality cell
2. In the VibeCheck admin interface, they assign a model to the new quality cell, or create a new model version in MLflow (for example for a new gearbox type)
3. During training, the TestStand user interface in the new quality cell shows the worker the normal manual acoustic testing routine, but starts recording data and creating a training data set
4. Gearbox motors marked as defective go to a repair cell, where repair technicians diagnose & repair the problem
5. Data from these diagnoses is imported into the VibeCheck dataset as well, to correct possible mislabeling by workers, and to increase the number of samples of defective motors
6. When enough data is available (at least 50 defective samples), a model is trained automatically in MLflow and the TestStand user interface begins showing prediction results
7. After some more validation with the worker, the model can work autonomously and only call in workers for defective or low-confidence results

Documentation and training materials will be created by the team for the first rollout, and improved during the first run by SEW.

Documentation includes:

Integration Requirements & Success Criteria

The MoSCoW method is a prioritization technique for requirements:

• Must have: Critical requirements that are non-negotiable for project success
• Should have: Important features that are highly desirable but not absolutely critical
• Could have: Nice-to-have features that would add value but can be deferred if necessary
• Won't have: Items explicitly excluded from the current scope but may be considered in future iterations

Phase 2 Summary

Roadmap

---
    config:
        gantt:
            barGap: 10
            fontSize: 18
            sectionFontSize: 18
            barHeight: 40
            leftPadding: 200
            numberSectionStyles: 2
---

gantt
    dateFormat  X
    axisFormat  Sprint %s
    tickInterval 1s

    section Infrastructure
    SEW#colon; Provide access to infrastructure    :crit, infra0, 0, 1s
    Simplify TestStand & LabVIEW integration  :infra4, 0, 2s
    Setup continuous learning infrastructure    :infra1, 0, 3s
    Setup model versioning & monitoring infrastructure    :infra2, 1, 1s
    IT Deployment at SEW           :infra3, 2, 1s

    section Machine Learning
    Improve data augmentation                  :ml1, 0, 1s
    Model versioning for gearbox types        :ml3, 0, 3s
    SEW#colon; Decide on further quality cell locations    :crit, ml0, 1, 1s
    Develop model validation & issue detection                     :ml2, after ml1, 2s
    Rollout to 2nd & 3rd cell             :vert, ml5, after ml2, 0
    Performance monitoring system             :ml4, after ml2, 1s
    Testing with 2nd gearbox type in 2nd cell            :ml6, after ml2, 2s
    Develop CV algorithm for sensor placement :ml7, 2, 1s

    section Deployment & Rollout
    Document rollout procedures              :deploy1, 0, 2s
    Create user handbook & UI info          :deploy2, after deploy1, 1s
    Support rollout to additional cells      :deploy3, after deploy1, 2s

Red tasks in the GANTT roadmap are tasks that require input from SEW:

Work packages

Rollen und Kosten

Hardware Kosten sind nicht Teil des Angebots.

Rate Card

Es gilt die Rate Card aus dem Rahmenvertrag, Stand 2025: