Model analysis report for

▶
root: {} 10 keys
- report_name: 'mlsimdriving_driving.analyse_lat_mlsim'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_lat_mlsim'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-harald_2025-06-15_15-45-45'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 12 keys
    - supercombo: 'f7db6a09-43c5-4db9-b856-7fe1a1c231eb/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1490 / 1658 aka 89.867%

better :1085 / 1658 aka 65.440%

best (0.5m):308 / 1658 aka 18.577%

global mae x: 0.227

global mae y: 0.268

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.245

avg resets: 0.305

Mean abs curvature: 0.211 deg/m

Mean abs curvature rate: 0.312 deg/ms

Mean abs lateral accel: 0.371 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.468 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :505 / 510 aka 99.020%

better :471 / 510 aka 92.353%

best (0.5m):190 / 510 aka 37.255%

global mae x: 0.158

global mae y: 0.203

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.200

avg resets: 0.022

Mean abs curvature: 0.044 deg/m

Mean abs curvature rate: 0.092 deg/ms

Mean abs lateral accel: 0.179 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.314 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :303 / 347 aka 87.320%

better :200 / 347 aka 57.637%

best (0.5m):62 / 347 aka 17.867%

global mae x: 0.300

global mae y: 0.296

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.270

avg resets: 0.510

Mean abs curvature: 0.418 deg/m

Mean abs curvature rate: 0.649 deg/ms

Mean abs lateral accel: 0.323 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.450 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :894 / 938 aka 95.309%

better :784 / 938 aka 83.582%

best (0.5m):271 / 938 aka 28.891%

global mae x: 0.178

global mae y: 0.231

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.215

avg resets: 0.085

Mean abs curvature: 0.114 deg/m

Mean abs curvature rate: 0.165 deg/ms

Mean abs lateral accel: 0.323 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.392 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :108 / 133 aka 81.203%

better :69 / 133 aka 51.880%

best (0.5m):17 / 133 aka 12.782%

global mae x: 0.255

global mae y: 0.317

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.284

avg resets: 0.248

Mean abs curvature: 0.275 deg/m

Mean abs curvature rate: 0.324 deg/ms

Mean abs lateral accel: 0.663 m/(ss)

Mean abs lateral accel GT: 0.645 m/(ss)

Mean abs lateral jerk: 0.609 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

104/120 aka 86.67%

better: 73/120 aka 60.83%

best (0.5m): 16/120 aka 13.33%

mean max error C3: 0.990618

mean max error C2: 1.209118

global mae: 0.290668

avg resets: 0.091667

Right lane change segments

135/150 aka 90.00%

better: 83/150 aka 55.33%

best (0.5m): 12/150 aka 8.00%

mean max error C3: 0.995012

mean max error C2: 1.202739

global mae: 0.278404

avg resets: 0.180000

Left turn segments

65/79 aka 82.28%

better: 30/79 aka 37.97%

best (0.5m): 3/79 aka 3.80%

mean max error C3: 1.326003

mean max error C2: 1.387215

global mae: 0.304584

avg resets: 0.886076

Right turn segments

75/91 aka 82.42%

better: 29/91 aka 31.87%

best (0.5m): 2/91 aka 2.20%

mean max error C3: 1.316842

mean max error C2: 1.425446

global mae: 0.334274

avg resets: 1.032967

Left keep segments

27/33 aka 81.82%

better: 15/33 aka 45.45%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.510489

mean max error C2: 1.219117

global mae: 0.285952

avg resets: 0.242424

Right keep segments

22/26 aka 84.62%

better: 14/26 aka 53.85%

best (0.5m): 1/26 aka 3.85%

mean max error C3: 1.262070

mean max error C2: 1.192100

global mae: 0.298640

avg resets: 0.115385

Plan y error in turns (> 6°/s)

Mean path error : -0.046 meters

Mean absolute path error : 0.478 meters

Turn bias test (y_pos at 50m > 3m)

got 29069 left samples

got 30815 right samples

mean left gt: -0.039733447

mean right gt: 0.0077234427

mean turn lanecenter error (GT pos, GT ll): left = 0.21726641058921814, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.18198084831237793, right = -0.24459317326545715

mean turn lanecenter error (GT pos, model ll): left = 0.2051863819360733, right = -0.25184062123298645

mean turn lanecenter error (sim pos, model ll): left = 0.17027057707309723, right = -0.21680933237075806

Mean abs curvature: 0.214301 deg/m

Mean abs curvature rate: 0.180950 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.172166 Hz, period 5.808350 s

Mean amplitude: 0.143098 m

Mean ping-pong significance (frequency x amplitude): 2.360293 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	179
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.736m
turn_right	+5.792m
lane_change_left	-1.431m
lane_change_right	+1.415m
keep_left	-0.783m
keep_right	+0.731m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.226s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.26s	3.04s	3.66s	4.10s
turn_right	2.27s	3.02s	3.63s	4.27s
lane_change_left	1.19s	1.77s	2.43s	3.86s
lane_change_right	1.14s	1.68s	2.35s	3.87s
keep_left	1.50s	2.31s	3.27s	4.84s
keep_right	1.75s	2.80s	3.79s	5.35s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.024	0.272
Easy straight	0.007	0.009	0.152
Low speed	0.017	0.036	0.378
No desire segments	0.008	0.014	0.212
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.366