Model analysis report for

▶
root: {} 10 keys
- report_name: 'rldriving_driving.analyse_lat_rl'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'rldriving_driving.analyse_lat_rl'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-06-17_09-59-38'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'SIMPLE'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 12 keys
    - supercombo: '967279c1-7d3c-4463-9d35-58e0311a5f57/700'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1569 / 1656 aka 94.746%

better :1197 / 1656 aka 72.283%

best (0.5m):517 / 1656 aka 31.220%

global mae x: 0.166

global mae y: 0.224

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.217

avg resets: 0.230

Mean abs curvature: 0.220 deg/m

Mean abs curvature rate: 0.304 deg/ms

Mean abs lateral accel: 0.372 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.488 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :507 / 508 aka 99.803%

better :497 / 508 aka 97.835%

best (0.5m):313 / 508 aka 61.614%

global mae x: 0.105

global mae y: 0.173

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.179

avg resets: 0.012

Mean abs curvature: 0.048 deg/m

Mean abs curvature rate: 0.086 deg/ms

Mean abs lateral accel: 0.176 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.312 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :325 / 347 aka 93.660%

better :217 / 347 aka 62.536%

best (0.5m):88 / 347 aka 25.360%

global mae x: 0.221

global mae y: 0.245

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.253

avg resets: 0.418

Mean abs curvature: 0.429 deg/m

Mean abs curvature rate: 0.610 deg/ms

Mean abs lateral accel: 0.317 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.449 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :919 / 936 aka 98.184%

better :832 / 936 aka 88.889%

best (0.5m):442 / 936 aka 47.222%

global mae x: 0.124

global mae y: 0.192

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.191

avg resets: 0.063

Mean abs curvature: 0.119 deg/m

Mean abs curvature rate: 0.164 deg/ms

Mean abs lateral accel: 0.322 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.401 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :120 / 132 aka 90.909%

better :76 / 132 aka 57.576%

best (0.5m):16 / 132 aka 12.121%

global mae x: 0.214

global mae y: 0.251

global mae y of lanecenter (when detected) in GT: 0.260

global mae y of lanecenter (when detected) in Sim: 0.240

avg resets: 0.197

Mean abs curvature: 0.290 deg/m

Mean abs curvature rate: 0.338 deg/ms

Mean abs lateral accel: 0.671 m/(ss)

Mean abs lateral accel GT: 0.649 m/(ss)

Mean abs lateral jerk: 0.651 m/(sss)

Mean abs lateral jerk GT: 0.398 m/(sss)

Left lane change segments

117/120 aka 97.50%

better: 91/120 aka 75.83%

best (0.5m): 26/120 aka 21.67%

mean max error C3: 0.712528

mean max error C2: 0.848257

global mae: 0.215301

avg resets: 0.033333

Right lane change segments

147/150 aka 98.00%

better: 110/150 aka 73.33%

best (0.5m): 32/150 aka 21.33%

mean max error C3: 0.839681

mean max error C2: 0.876606

global mae: 0.230219

avg resets: 0.100000

Left turn segments

68/79 aka 86.08%

better: 25/79 aka 31.65%

best (0.5m): 2/79 aka 2.53%

mean max error C3: 1.335308

mean max error C2: 1.365299

global mae: 0.281250

avg resets: 0.721519

Right turn segments

84/91 aka 92.31%

better: 35/91 aka 38.46%

best (0.5m): 3/91 aka 3.30%

mean max error C3: 1.164642

mean max error C2: 1.249053

global mae: 0.273511

avg resets: 0.769231

Left keep segments

27/33 aka 81.82%

better: 13/33 aka 39.39%

best (0.5m): 2/33 aka 6.06%

mean max error C3: 1.478981

mean max error C2: 1.318321

global mae: 0.256391

avg resets: 0.181818

Right keep segments

22/26 aka 84.62%

better: 13/26 aka 50.00%

best (0.5m): 2/26 aka 7.69%

mean max error C3: 1.403938

mean max error C2: 1.179456

global mae: 0.270926

avg resets: 0.153846

Plan y error in turns (> 6°/s)

Mean path error : -0.020 meters

Mean absolute path error : 0.428 meters

Turn bias test (y_pos at 50m > 3m)

got 29069 left samples

got 30815 right samples

mean left gt: -0.03801057

mean right gt: -0.0031960653

mean turn lanecenter error (GT pos, GT ll): left = 0.21726641058921814, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.1968015432357788, right = -0.2592889368534088

mean turn lanecenter error (GT pos, model ll): left = 0.21546684205532074, right = -0.2508232295513153

mean turn lanecenter error (sim pos, model ll): left = 0.17283715307712555, right = -0.24063949286937714

Mean abs curvature: 0.214153 deg/m

Mean abs curvature rate: 0.205625 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.173369 Hz, period 5.768039 s

Mean amplitude: 0.122271 m

Mean ping-pong significance (frequency x amplitude): 2.040486 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	179
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-0.434m
turn_right	+0.630m
lane_change_left	-1.628m
lane_change_right	+1.542m
keep_left	-0.448m
keep_right	+0.734m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.226s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	3.31s	4.28s	4.84s	5.53s
turn_right	3.31s	4.12s	4.83s	5.72s
lane_change_left	0.51s	1.25s	2.12s	3.59s
lane_change_right	0.68s	1.34s	2.27s	3.76s
keep_left	3.51s	4.78s	6.25s	7.51s
keep_right	3.52s	4.69s	5.52s	6.73s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.265
Easy straight	0.006	0.009	0.144
Low speed	0.017	0.035	0.372
No desire segments	0.007	0.014	0.205
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.362