Model analysis report for

• ▶
  root: {} 9 keys
  • report_name: 'rldriving_analyselatcheat'
  • output_dir: null
  • save_tmp: false
  • read_only: false
  • limit: 10000
  • format: 'HTML'
  • ▶
    miniray: {} 8 keys
    • priority: 1000
    • job_name: 'rldriving_analyselatcheat'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: 'caches/workstation-yassine_2025-01-25_18-29-03'
    • force_local: false
  • route_list: '/home/batman/xx/routes/default_test_2k'
  • ▶
    rollout: {} 10 keys
    • type: 'TinySimRL'
    • nowindow: false
    • debug: false
    • debug_render: false
    • save_observation_keys: []
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 24 keys
      • ▶
        ep: {} 6 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
        • ▶
          input_paths: {} 0 keys
      • start: null
      • end: null
      • get_imgs: true
      • allow_resets: true
      • zero_desire: true
      • fps: 20
      • compressor_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • lpips_model: null
      • future_observation_range_seconds: null
      • max_x_error: 4.0
      • max_y_error: 3.0
      • max_yaw_error: 0.2
      • reset_every: null
      • noise_model: 'REALISTIC'
      • noise_end: null
      • photo_noise_model: 'NONE'
      • augment_mode: 'DEPTH'
      • inpaint_mode: 'NONE'
      • avoid_hood: false
      • cache_granularity: true
      • worldmodel: null
    • ▶
      agent: {} 15 keys
      • supercombo: 'b25c27a8-263b-4f3f-b1f8-72ac49b4f9c7/700'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • act_on_lat: true
      • act_on_long: false
      • min_hist_lat: true
      • min_hist_long: 10
      • calibrate_speed: false
      • run_planners: true
      • use_long_mpc: true
      • use_allow_throttle: false
      • use_prev_desired_curv: true
      • model_trained_fps: 20
      • use_worldmodel_as_agent: false
      • use_raw_plan: false
    • ▶
      shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1295 / 1657 aka 78.153%

better :939 / 1657 aka 56.669%

best (0.5m):419 / 1657 aka 25.287%

global mae x: 0.200

global mae y: 0.262

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.261

avg resets: 0.482

Mean abs curvature: 0.527 deg/m

Mean abs curvature rate: 0.875 deg/ms

Mean abs lateral accel: 0.381 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.601 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :507 / 510 aka 99.412%

better :489 / 510 aka 95.882%

best (0.5m):272 / 510 aka 53.333%

global mae x: 0.138

global mae y: 0.180

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.194

avg resets: 0.024

Mean abs curvature: 0.434 deg/m

Mean abs curvature rate: 0.656 deg/ms

Mean abs lateral accel: 0.183 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.356 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :308 / 347 aka 88.761%

better :183 / 347 aka 52.738%

best (0.5m):75 / 347 aka 21.614%

global mae x: 0.261

global mae y: 0.264

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.267

avg resets: 0.726

Mean abs curvature: 0.915 deg/m

Mean abs curvature rate: 1.576 deg/ms

Mean abs lateral accel: 0.324 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.531 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :922 / 938 aka 98.294%

better :819 / 938 aka 87.313%

best (0.5m):403 / 938 aka 42.964%

global mae x: 0.153

global mae y: 0.199

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.202

avg resets: 0.084

Mean abs curvature: 0.444 deg/m

Mean abs curvature rate: 0.655 deg/ms

Mean abs lateral accel: 0.327 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.455 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :124 / 132 aka 93.939%

better :77 / 132 aka 58.333%

best (0.5m):15 / 132 aka 11.364%

global mae x: 0.244

global mae y: 0.255

global mae y of lanecenter (when detected) in GT: 0.260

global mae y of lanecenter (when detected) in Sim: 0.238

avg resets: 0.152

Mean abs curvature: 0.687 deg/m

Mean abs curvature rate: 1.008 deg/ms

Mean abs lateral accel: 0.673 m/(ss)

Mean abs lateral accel GT: 0.649 m/(ss)

Mean abs lateral jerk: 0.736 m/(sss)

Mean abs lateral jerk GT: 0.398 m/(sss)

Left lane change segments

44/120 aka 36.67%

better: 10/120 aka 8.33%

best (0.5m): 4/120 aka 3.33%

mean max error C3: 2.108435

mean max error C2: 2.255231

global mae: 0.315849

avg resets: 0.416667

Right lane change segments

44/150 aka 29.33%

better: 18/150 aka 12.00%

best (0.5m): 4/150 aka 2.67%

mean max error C3: 2.207100

mean max error C2: 2.347128

global mae: 0.358492

avg resets: 0.573333

Left turn segments

65/79 aka 82.28%

better: 24/79 aka 30.38%

best (0.5m): 2/79 aka 2.53%

mean max error C3: 1.450405

mean max error C2: 1.386952

global mae: 0.287543

avg resets: 1.582278

Right turn segments

76/90 aka 84.44%

better: 29/90 aka 32.22%

best (0.5m): 2/90 aka 2.22%

mean max error C3: 1.267675

mean max error C2: 1.404525

global mae: 0.298497

avg resets: 1.155556

Left keep segments

23/33 aka 69.70%

better: 11/33 aka 33.33%

best (0.5m): 3/33 aka 9.09%

mean max error C3: 1.620491

mean max error C2: 1.576152

global mae: 0.277369

avg resets: 0.515152

Right keep segments

21/26 aka 80.77%

better: 8/26 aka 30.77%

best (0.5m): 1/26 aka 3.85%

mean max error C3: 1.738087

mean max error C2: 1.173964

global mae: 0.300492

avg resets: 0.153846

Plan y error in turns (> 6°/s)

Mean path error : -0.007 meters

Mean absolute path error : 0.457 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: -0.02163109

mean right gt: 0.0001048343

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.20254455506801605, right = -0.2594310939311981

mean turn lanecenter error (GT pos, model ll): left = 0.18116335570812225, right = -0.215867817401886

mean turn lanecenter error (sim pos, model ll): left = 0.16992296278476715, right = -0.20128262042999268

Mean abs curvature: 0.215361 deg/m

Mean abs curvature rate: 0.266047 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.173654 Hz, period 5.758594 s

Mean amplitude: 0.142763 m

Mean ping-pong significance (frequency x amplitude): 2.457829 %