Model analysis report for

▶
root: {} 10 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-weixing2_2025-03-06_21-31-46'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 25 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - lpips_model: null
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - noise_end: null
    - photo_noise_model: 'NONE'
    - augment_mode: 'SIMPLE'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 15 keys
    - supercombo: 'e07ce1de-bdea-463e-b5bc-a38ce8d43f4f/700'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - act_on_lat: true
    - act_on_long: false
    - min_hist_lat: true
    - min_hist_long: 10
    - calibrate_speed: false
    - run_planners: true
    - use_long_mpc: true
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1580 / 1657 aka 95.353%

better :1162 / 1657 aka 70.127%

best (0.5m):438 / 1657 aka 26.433%

global mae x: 0.196

global mae y: 0.232

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.223

avg resets: 0.291

Mean abs curvature: 0.222 deg/m

Mean abs curvature rate: 0.336 deg/ms

Mean abs lateral accel: 0.373 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.520 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :510 / 510 aka 100.000%

better :489 / 510 aka 95.882%

best (0.5m):273 / 510 aka 53.529%

global mae x: 0.129

global mae y: 0.180

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.188

avg resets: 0.016

Mean abs curvature: 0.048 deg/m

Mean abs curvature rate: 0.094 deg/ms

Mean abs lateral accel: 0.173 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.321 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :328 / 347 aka 94.524%

better :211 / 347 aka 60.807%

best (0.5m):72 / 347 aka 20.749%

global mae x: 0.262

global mae y: 0.248

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.247

avg resets: 0.539

Mean abs curvature: 0.430 deg/m

Mean abs curvature rate: 0.673 deg/ms

Mean abs lateral accel: 0.322 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.494 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :921 / 938 aka 98.188%

better :820 / 938 aka 87.420%

best (0.5m):381 / 938 aka 40.618%

global mae x: 0.149

global mae y: 0.201

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.199

avg resets: 0.082

Mean abs curvature: 0.119 deg/m

Mean abs curvature rate: 0.183 deg/ms

Mean abs lateral accel: 0.320 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.419 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :122 / 132 aka 92.424%

better :76 / 132 aka 57.576%

best (0.5m):14 / 132 aka 10.606%

global mae x: 0.252

global mae y: 0.261

global mae y of lanecenter (when detected) in GT: 0.260

global mae y of lanecenter (when detected) in Sim: 0.238

avg resets: 0.227

Mean abs curvature: 0.293 deg/m

Mean abs curvature rate: 0.371 deg/ms

Mean abs lateral accel: 0.672 m/(ss)

Mean abs lateral accel GT: 0.649 m/(ss)

Mean abs lateral jerk: 0.702 m/(sss)

Mean abs lateral jerk GT: 0.398 m/(sss)

Left lane change segments

117/120 aka 97.50%

better: 91/120 aka 75.83%

best (0.5m): 28/120 aka 23.33%

mean max error C3: 0.726312

mean max error C2: 0.865517

global mae: 0.220042

avg resets: 0.041667

Right lane change segments

141/150 aka 94.00%

better: 99/150 aka 66.00%

best (0.5m): 17/150 aka 11.33%

mean max error C3: 0.894414

mean max error C2: 0.998905

global mae: 0.242451

avg resets: 0.133333

Left turn segments

73/79 aka 92.41%

better: 30/79 aka 37.97%

best (0.5m): 1/79 aka 1.27%

mean max error C3: 1.295572

mean max error C2: 1.185271

global mae: 0.270920

avg resets: 0.886076

Right turn segments

86/90 aka 95.56%

better: 31/90 aka 34.44%

best (0.5m): 4/90 aka 4.44%

mean max error C3: 1.200352

mean max error C2: 1.265469

global mae: 0.279714

avg resets: 0.922222

Left keep segments

27/33 aka 81.82%

better: 10/33 aka 30.30%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.184493

mean max error C2: 1.458560

global mae: 0.257040

avg resets: 0.303030

Right keep segments

23/26 aka 88.46%

better: 12/26 aka 46.15%

best (0.5m): 0/26 aka 0.00%

mean max error C3: 1.204787

mean max error C2: 1.173416

global mae: 0.289524

avg resets: 0.115385

Plan y error in turns (> 6°/s)

Mean path error : -0.010 meters

Mean absolute path error : 0.439 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: -0.008566487

mean right gt: 0.006802595

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.21474789083003998, right = -0.24396757781505585

mean turn lanecenter error (GT pos, model ll): left = 0.21189159154891968, right = -0.2272624671459198

mean turn lanecenter error (sim pos, model ll): left = 0.18914559483528137, right = -0.19927048683166504

Mean abs curvature: 0.213775 deg/m

Mean abs curvature rate: 0.217033 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.164418 Hz, period 6.082042 s

Mean amplitude: 0.127057 m

Mean ping-pong significance (frequency x amplitude): 2.028116 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	178
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-1.560m
turn_right	+0.544m
lane_change_left	-1.644m
lane_change_right	+1.586m
keep_left	-0.534m
keep_right	+0.729m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.235s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	3.23s	3.98s	4.51s	5.26s
turn_right	3.06s	4.10s	5.02s	5.68s
lane_change_left	0.57s	1.21s	2.11s	3.60s
lane_change_right	0.55s	1.16s	2.09s	3.61s
keep_left	3.59s	5.10s	6.22s	7.96s
keep_right	3.05s	4.86s	6.21s	6.98s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.262
Easy straight	0.006	0.009	0.139
Low speed	0.017	0.036	0.370
No desire segments	0.007	0.014	0.202
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.363