Tutorial 18: Monte Carlo Verification

This tutorial demonstrates how to use the Monte Carlo Simulation harness and the statistical analyzer for GNC system verification.

1. Overview

For mission-critical spacecraft software, a single simulation run is insufficient to prove robustness. OpenGNC provides a high-performance Monte Carlo suite to analyze performance under stochastic variations in environment, sensor noise, and process disturbances.

2. Running a Monte Carlo Simulation

The MonteCarloSim class facilitates parallel execution of multiple trials.

[!IMPORTANT] Windows & Jupyter Note: To avoid hangs when using multiprocessing on Windows, simulator functions must be defined in a separate .py file and imported. This allows child processes to cleanly import the function.

from opengnc.simulation.monte_carlo import MonteCarloSim
from mc_worker import fast_simulator  # Imported from external file for Windows compatibility
import numpy as np
import time

mc = MonteCarloSim(fast_simulator)

# Execute 100 trials using all available CPU cores
print("Starting parallel Monte Carlo (100 runs)...")
start = time.time()
results = mc.run_parallel(num_runs=100, tf=10.0, dt=0.01)
end = time.time()

print(f"Completed {len(results)} trials in {end - start:.2f} seconds.")

Starting parallel Monte Carlo (100 runs)...
Completed 100 trials in 0.01 seconds.

3. Statistical Analysis & Verification

Once the trials are complete, you can use the MonteCarloAnalyzer to calculate performance margins.

analyzer = mc.get_analyzer()

# 1. Calculate 3-Sigma Bounds
pos_stats = analyzer.get_aggregate_stats("pos_error")

print(f"Final Mean Error: {pos_stats['mean'][-1]:.4f} m")
print(f"Final 1-Sigma: {pos_stats['std'][-1]:.4f} m")
print(f"3-Sigma Upper Bound: {pos_stats['sigma_3_upper'][-1]:.4f} m")

# 2. Reliability Analysis (criteria: error < 2.0m)
failure_func = lambda res: np.any(np.abs(res["pos_error"]) > 2.0)
summary = analyzer.summarize_failures(failure_func)

print(f"Maximum allowable error: 2.0m")
print(f"Successful Trials: {summary['total_runs'] - summary['failures']} / {summary['total_runs']}")
print(f"Reliability Score: {summary['reliability'] * 100:.1f}%")

Final Mean Error: 1.0030 m
Final 1-Sigma: 0.0546 m
3-Sigma Upper Bound: 1.1668 m
Maximum allowable error: 2.0m
Successful Trials: 100 / 100
Reliability Score: 100.0%

4. Verification Suite Tool

A pre-configured verification suite is available at benchmarks/run_verification.py for standard proof-of-mission analysis.

python benchmarks/run_verification.py