Model analysis report for

▶
root: {} 10 keys
- report_name: 'mlsimdriving_driving.analyse_lat_mlsim'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_lat_mlsim'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-07-06_10-53-22'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 32 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'COROLLA'
      - ▶
        ml_model_config: {} 6 keys
        
        platform: 'COROLLA'
        
        eid: 'd3e5ef06-14ec-4e35-b708-d8db73d29f1e/51'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 12 keys
    - supercombo: 'fef4d896-597e-4c6f-90cf-8eb7102c89d2/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1447 / 1644 aka 88.017%

better :1009 / 1644 aka 61.375%

best (0.5m):275 / 1644 aka 16.727%

global mae x: 0.170

global mae y: 0.291

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.263

avg resets: 0.262

Mean abs curvature: 0.201 deg/m

Mean abs curvature rate: 0.271 deg/ms

Mean abs lateral accel: 0.359 m/(ss)

Mean abs lateral accel GT: 0.337 m/(ss)

Mean abs lateral jerk: 0.396 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :496 / 505 aka 98.218%

better :462 / 505 aka 91.485%

best (0.5m):177 / 505 aka 35.050%

global mae x: 0.106

global mae y: 0.215

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.213

avg resets: 0.018

Mean abs curvature: 0.040 deg/m

Mean abs curvature rate: 0.074 deg/ms

Mean abs lateral accel: 0.170 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.268 m/(sss)

Mean abs lateral jerk GT: 0.170 m/(sss)

Low speed

Metrics

passed :285 / 347 aka 82.133%

better :165 / 347 aka 47.550%

best (0.5m):55 / 347 aka 15.850%

global mae x: 0.234

global mae y: 0.333

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.302

avg resets: 0.461

Mean abs curvature: 0.396 deg/m

Mean abs curvature rate: 0.557 deg/ms

Mean abs lateral accel: 0.307 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.364 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :884 / 933 aka 94.748%

better :752 / 933 aka 80.600%

best (0.5m):245 / 933 aka 26.259%

global mae x: 0.127

global mae y: 0.244

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.228

avg resets: 0.084

Mean abs curvature: 0.109 deg/m

Mean abs curvature rate: 0.143 deg/ms

Mean abs lateral accel: 0.313 m/(ss)

Mean abs lateral accel GT: 0.285 m/(ss)

Mean abs lateral jerk: 0.336 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :111 / 132 aka 84.091%

better :64 / 132 aka 48.485%

best (0.5m):7 / 132 aka 5.303%

global mae x: 0.217

global mae y: 0.316

global mae y of lanecenter (when detected) in GT: 0.262

global mae y of lanecenter (when detected) in Sim: 0.288

avg resets: 0.235

Mean abs curvature: 0.271 deg/m

Mean abs curvature rate: 0.285 deg/ms

Mean abs lateral accel: 0.649 m/(ss)

Mean abs lateral accel GT: 0.649 m/(ss)

Mean abs lateral jerk: 0.522 m/(sss)

Mean abs lateral jerk GT: 0.398 m/(sss)

Left lane change segments

109/119 aka 91.60%

better: 67/119 aka 56.30%

best (0.5m): 10/119 aka 8.40%

mean max error C3: 1.028679

mean max error C2: 1.112398

global mae: 0.274444

avg resets: 0.067227

Right lane change segments

130/148 aka 87.84%

better: 72/148 aka 48.65%

best (0.5m): 11/148 aka 7.43%

mean max error C3: 1.130422

mean max error C2: 1.248419

global mae: 0.314355

avg resets: 0.175676

Left turn segments

56/78 aka 71.79%

better: 20/78 aka 25.64%

best (0.5m): 2/78 aka 2.56%

mean max error C3: 1.650067

mean max error C2: 1.594506

global mae: 0.376168

avg resets: 0.833333

Right turn segments

71/91 aka 78.02%

better: 30/91 aka 32.97%

best (0.5m): 2/91 aka 2.20%

mean max error C3: 1.283743

mean max error C2: 1.530382

global mae: 0.370675

avg resets: 0.813187

Left keep segments

25/32 aka 78.12%

better: 12/32 aka 37.50%

best (0.5m): 0/32 aka 0.00%

mean max error C3: 1.575444

mean max error C2: 1.334862

global mae: 0.327133

avg resets: 0.218750

Right keep segments

18/26 aka 69.23%

better: 9/26 aka 34.62%

best (0.5m): 1/26 aka 3.85%

mean max error C3: 1.521362

mean max error C2: 1.476320

global mae: 0.351714

avg resets: 0.307692

Plan y error in turns (> 6°/s)

Mean path error : 0.067 meters

Mean absolute path error : 0.525 meters

Turn bias test (y_pos at 50m > 3m)

got 29042 left samples

got 30742 right samples

mean left gt: 0.011186116

mean right gt: 0.05657463

mean turn lanecenter error (GT pos, GT ll): left = 0.21758385002613068, right = -0.24763436615467072

mean turn lanecenter error (sim pos, GT ll): left = 0.24449050426483154, right = -0.16783662140369415

mean turn lanecenter error (GT pos, model ll): left = 0.21493692696094513, right = -0.24576562643051147

mean turn lanecenter error (sim pos, model ll): left = 0.19142819941043854, right = -0.15496765077114105

Mean abs curvature: 0.212792 deg/m

Mean abs curvature rate: 0.166207 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.168433 Hz, period 5.937092 s

Mean amplitude: 0.145549 m

Mean ping-pong significance (frequency x amplitude): 2.339881 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	164
turn_right	178
lane_change_left	231
lane_change_right	279
keep_left	79
keep_right	66

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-6.376m
turn_right	+5.885m
lane_change_left	-1.417m
lane_change_right	+1.341m
keep_left	-0.714m
keep_right	+1.085m

Mean desire duration - ground truth

Command	Duration
turn_left	6.763s
turn_right	6.261s
lane_change_left	3.305s
lane_change_right	3.493s
keep_left	3.746s
keep_right	3.848s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.23s	2.93s	3.45s	4.14s
turn_right	2.43s	3.20s	3.75s	4.51s
lane_change_left	1.23s	1.80s	2.58s	4.10s
lane_change_right	1.26s	1.86s	2.62s	4.21s
keep_left	1.61s	2.41s	3.33s	5.40s
keep_right	1.83s	2.76s	3.72s	5.36s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.284
Easy straight	0.007	0.009	0.162
Low speed	0.018	0.037	0.386
No desire segments	0.008	0.014	0.225
No desire turns (max lat accel > 2.5 m/ss)	0.013	0.029	0.380