Model analysis report for

• ▶
  root: {} 10 keys
  • report_name: 'rldriving_analyselatcheat'
  • output_dir: null
  • save_tmp: false
  • read_only: false
  • limit: 10000
  • use_virtual_webgl: false
  • format: 'HTML'
  • ▶
    miniray: {} 8 keys
    • priority: 1000
    • job_name: 'rldriving_analyselatcheat'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: 'caches/workstation-weixing2_2025-03-06_21-31-46'
    • force_local: false
  • route_list: '/home/batman/xx/routes/default_test_2k'
  • ▶
    rollout: {} 10 keys
    • type: 'TinySimRL'
    • nowindow: false
    • debug: false
    • debug_render: false
    • save_observation_keys: []
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 25 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
        • feature_dir: null
        • ▶
          input_paths: {} 0 keys
      • start: null
      • end: null
      • get_imgs: true
      • allow_resets: true
      • zero_desire: true
      • fps: 20
      • compressor_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • lpips_model: null
      • future_observation_range_seconds: null
      • max_x_error: 4.0
      • max_y_error: 3.0
      • max_yaw_error: 0.2
      • reset_every: null
      • noise_model: 'REALISTIC'
      • noise_end: null
      • photo_noise_model: 'NONE'
      • augment_mode: 'DEPTH'
      • inpaint_mode: 'NONE'
      • avoid_hood: false
      • cache_granularity: true
      • worldmodel: null
    • ▶
      agent: {} 15 keys
      • supercombo: 'e07ce1de-bdea-463e-b5bc-a38ce8d43f4f/700'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • act_on_lat: true
      • act_on_long: false
      • min_hist_lat: true
      • min_hist_long: 10
      • calibrate_speed: false
      • run_planners: true
      • use_long_mpc: true
      • use_allow_throttle: false
      • use_prev_desired_curv: true
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan: false
    • ▶
      shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1278 / 1657 aka 77.127%

better :898 / 1657 aka 54.194%

best (0.5m):380 / 1657 aka 22.933%

global mae x: 0.198

global mae y: 0.270

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.260

avg resets: 0.416

Mean abs curvature: 0.217 deg/m

Mean abs curvature rate: 0.351 deg/ms

Mean abs lateral accel: 0.372 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.540 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :506 / 510 aka 99.216%

better :484 / 510 aka 94.902%

best (0.5m):268 / 510 aka 52.549%

global mae x: 0.128

global mae y: 0.182

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.192

avg resets: 0.016

Mean abs curvature: 0.049 deg/m

Mean abs curvature rate: 0.102 deg/ms

Mean abs lateral accel: 0.173 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.315 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :300 / 347 aka 86.455%

better :183 / 347 aka 52.738%

best (0.5m):67 / 347 aka 19.308%

global mae x: 0.264

global mae y: 0.276

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.274

avg resets: 0.588

Mean abs curvature: 0.419 deg/m

Mean abs curvature rate: 0.691 deg/ms

Mean abs lateral accel: 0.318 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.492 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :912 / 938 aka 97.228%

better :803 / 938 aka 85.608%

best (0.5m):369 / 938 aka 39.339%

global mae x: 0.150

global mae y: 0.205

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.201

avg resets: 0.069

Mean abs curvature: 0.120 deg/m

Mean abs curvature rate: 0.186 deg/ms

Mean abs lateral accel: 0.318 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.408 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :120 / 133 aka 90.226%

better :69 / 133 aka 51.880%

best (0.5m):13 / 133 aka 9.774%

global mae x: 0.257

global mae y: 0.271

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.241

avg resets: 0.211

Mean abs curvature: 0.293 deg/m

Mean abs curvature rate: 0.378 deg/ms

Mean abs lateral accel: 0.665 m/(ss)

Mean abs lateral accel GT: 0.646 m/(ss)

Mean abs lateral jerk: 0.677 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

39/120 aka 32.50%

better: 10/120 aka 8.33%

best (0.5m): 3/120 aka 2.50%

mean max error C3: 2.042292

mean max error C2: 2.372292

global mae: 0.323680

avg resets: 0.333333

Right lane change segments

41/150 aka 27.33%

better: 12/150 aka 8.00%

best (0.5m): 3/150 aka 2.00%

mean max error C3: 2.222206

mean max error C2: 2.434669

global mae: 0.360453

avg resets: 0.600000

Left turn segments

68/79 aka 86.08%

better: 22/79 aka 27.85%

best (0.5m): 1/79 aka 1.27%

mean max error C3: 1.305354

mean max error C2: 1.534944

global mae: 0.298217

avg resets: 0.974684

Right turn segments

81/90 aka 90.00%

better: 20/90 aka 22.22%

best (0.5m): 2/90 aka 2.22%

mean max error C3: 1.185360

mean max error C2: 1.450705

global mae: 0.303569

avg resets: 0.955556

Left keep segments

20/33 aka 60.61%

better: 8/33 aka 24.24%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.465062

mean max error C2: 1.771784

global mae: 0.284422

avg resets: 0.484848

Right keep segments

19/26 aka 73.08%

better: 6/26 aka 23.08%

best (0.5m): 0/26 aka 0.00%

mean max error C3: 1.945632

mean max error C2: 1.464820

global mae: 0.308556

avg resets: 0.230769

Plan y error in turns (> 6°/s)

Mean path error : -0.006 meters

Mean absolute path error : 0.448 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: 0.0012421451

mean right gt: 0.005779906

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762772023677826

mean turn lanecenter error (sim pos, GT ll): left = 0.2387758493423462, right = -0.2521309554576874

mean turn lanecenter error (GT pos, model ll): left = 0.18147289752960205, right = -0.23151853680610657

mean turn lanecenter error (sim pos, model ll): left = 0.18954025208950043, right = -0.1933249682188034

Mean abs curvature: 0.214201 deg/m

Mean abs curvature rate: 0.221784 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.166607 Hz, period 6.002139 s

Mean amplitude: 0.140430 m

Mean ping-pong significance (frequency x amplitude): 2.258203 %