Model analysis report for

▶
root: {} 9 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-01-25_18-29-03'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 24 keys
    - ▶
      ep: {} 6 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - lpips_model: null
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - noise_end: null
    - photo_noise_model: 'NONE'
    - augment_mode: 'SIMPLE'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 15 keys
    - supercombo: 'b25c27a8-263b-4f3f-b1f8-72ac49b4f9c7/700'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - act_on_lat: true
    - act_on_long: false
    - min_hist_lat: true
    - min_hist_long: 10
    - calibrate_speed: false
    - run_planners: true
    - use_long_mpc: true
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 20
    - use_worldmodel_as_agent: false
    - use_raw_plan: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1569 / 1657 aka 94.689%

better :1193 / 1657 aka 71.998%

best (0.5m):477 / 1657 aka 28.787%

global mae x: 0.200

global mae y: 0.230

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.226

avg resets: 0.345

Mean abs curvature: 0.521 deg/m

Mean abs curvature rate: 0.769 deg/ms

Mean abs lateral accel: 0.382 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.562 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :509 / 510 aka 99.804%

better :490 / 510 aka 96.078%

best (0.5m):271 / 510 aka 53.137%

global mae x: 0.137

global mae y: 0.182

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.194

avg resets: 0.022

Mean abs curvature: 0.427 deg/m

Mean abs curvature rate: 0.517 deg/ms

Mean abs lateral accel: 0.184 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.356 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :324 / 347 aka 93.372%

better :201 / 347 aka 57.925%

best (0.5m):86 / 347 aka 24.784%

global mae x: 0.261

global mae y: 0.250

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.256

avg resets: 0.625

Mean abs curvature: 0.911 deg/m

Mean abs curvature rate: 1.405 deg/ms

Mean abs lateral accel: 0.327 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.526 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :922 / 938 aka 98.294%

better :829 / 938 aka 88.380%

best (0.5m):404 / 938 aka 43.070%

global mae x: 0.153

global mae y: 0.199

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.203

avg resets: 0.086

Mean abs curvature: 0.440 deg/m

Mean abs curvature rate: 0.538 deg/ms

Mean abs lateral accel: 0.328 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.458 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :124 / 132 aka 93.939%

better :81 / 132 aka 61.364%

best (0.5m):18 / 132 aka 13.636%

global mae x: 0.244

global mae y: 0.256

global mae y of lanecenter (when detected) in GT: 0.260

global mae y of lanecenter (when detected) in Sim: 0.241

avg resets: 0.152

Mean abs curvature: 0.694 deg/m

Mean abs curvature rate: 0.803 deg/ms

Mean abs lateral accel: 0.678 m/(ss)

Mean abs lateral accel GT: 0.649 m/(ss)

Mean abs lateral jerk: 0.747 m/(sss)

Mean abs lateral jerk GT: 0.398 m/(sss)

Left lane change segments

117/120 aka 97.50%

better: 97/120 aka 80.83%

best (0.5m): 27/120 aka 22.50%

mean max error C3: 0.677455

mean max error C2: 0.850227

global mae: 0.219422

avg resets: 0.075000

Right lane change segments

144/150 aka 96.00%

better: 109/150 aka 72.67%

best (0.5m): 27/150 aka 18.00%

mean max error C3: 0.858724

mean max error C2: 0.925698

global mae: 0.241351

avg resets: 0.160000

Left turn segments

67/79 aka 84.81%

better: 24/79 aka 30.38%

best (0.5m): 1/79 aka 1.27%

mean max error C3: 1.380808

mean max error C2: 1.395282

global mae: 0.286562

avg resets: 1.341772

Right turn segments

79/90 aka 87.78%

better: 28/90 aka 31.11%

best (0.5m): 4/90 aka 4.44%

mean max error C3: 1.190229

mean max error C2: 1.380269

global mae: 0.286374

avg resets: 1.011111

Left keep segments

29/33 aka 87.88%

better: 11/33 aka 33.33%

best (0.5m): 3/33 aka 9.09%

mean max error C3: 1.517369

mean max error C2: 1.172623

global mae: 0.255193

avg resets: 0.363636

Right keep segments

25/26 aka 96.15%

better: 16/26 aka 61.54%

best (0.5m): 0/26 aka 0.00%

mean max error C3: 1.168333

mean max error C2: 1.008996

global mae: 0.281273

avg resets: 0.153846

Plan y error in turns (> 6°/s)

Mean path error : -0.008 meters

Mean absolute path error : 0.446 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: -0.034606002

mean right gt: -0.011403198

mean turn lanecenter error (GT pos, GT ll): left = 0.21541057527065277, right = -0.24762772023677826

mean turn lanecenter error (sim pos, GT ll): left = 0.18718960881233215, right = -0.2706342935562134

mean turn lanecenter error (GT pos, model ll): left = 0.20516954362392426, right = -0.2190309464931488

mean turn lanecenter error (sim pos, model ll): left = 0.1704849749803543, right = -0.21433258056640625

Mean abs curvature: 0.215171 deg/m

Mean abs curvature rate: 0.258148 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.175772 Hz, period 5.689199 s

Mean amplitude: 0.125493 m

Mean ping-pong significance (frequency x amplitude): 2.145620 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	178
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-1.386m
turn_right	+1.535m
lane_change_left	-1.619m
lane_change_right	+1.570m
keep_left	-0.370m
keep_right	+0.520m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.235s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	3.16s	4.10s	4.87s	5.58s
turn_right	3.27s	4.23s	4.74s	5.46s
lane_change_left	0.56s	1.25s	2.12s	3.77s
lane_change_right	0.62s	1.30s	2.21s	3.83s
keep_left	3.17s	4.43s	5.83s	7.38s
keep_right	3.78s	5.37s	6.38s	7.58s

Group	Gas prob	Brake prob	Steer prob
All segments	0.009	0.022	0.271
Easy straight	0.006	0.008	0.147
Low speed	0.016	0.035	0.374
No desire segments	0.007	0.014	0.210
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.027	0.372