Model analysis report for

• ▶
  root: {} 11 keys
  • report_name: 'rldriving_analyselatcheat'
  • output_dir: null
  • save_tmp: false
  • read_only: false
  • limit: 10000
  • shuffle: false
  • use_virtual_webgl: false
  • format: 'HTML'
  • ▶
    miniray: {} 8 keys
    • priority: 1000
    • job_name: 'rldriving_analyselatcheat'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: 'caches/workstation-yassine_2025-05-13_10-13-43'
    • force_local: false
  • route_list: '/home/batman/xx/routes/default_test_2k'
  • ▶
    rollout: {} 10 keys
    • type: 'TinySimRL'
    • nowindow: false
    • debug: false
    • debug_render: false
    • save_observation_keys: []
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 30 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
        • feature_dir: null
        • ▶
          input_paths: {} 0 keys
      • start: null
      • end: null
      • env: 'xx.arena.envs.tinysim_env.TinySimEnv'
      • get_imgs: true
      • allow_resets: true
      • zero_desire: true
      • fps: 20
      • allow_lat: true
      • allow_long: false
      • min_hist: 40
      • compressor_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • yolo_model: false
      • lpips_model: false
      • sharpness_metric: true
      • future_observation_range_seconds: null
      • max_x_error: 4.0
      • max_y_error: 3.0
      • max_yaw_error: 0.2
      • reset_every: null
      • noise_model: 'REALISTIC'
      • photo_noise_model: 'NONE'
      • augment_mode: 'DEPTH'
      • inpaint_mode: 'NONE'
      • avoid_hood: false
      • cache_granularity: true
      • worldmodel: null
    • ▶
      agent: {} 10 keys
      • supercombo: 'e30305e9-5fb8-48bd-a278-f714e880aef1/700'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • calibrate_speed: false
      • use_allow_throttle: false
      • use_prev_desired_curv: true
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan_lat: false
      • use_raw_plan_long: false
    • ▶
      shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1259 / 1658 aka 75.935%

better :915 / 1658 aka 55.187%

best (0.5m):388 / 1658 aka 23.402%

global mae x: 0.228

global mae y: 0.266

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.257

avg resets: 0.420

Mean abs curvature: 0.224 deg/m

Mean abs curvature rate: 0.322 deg/ms

Mean abs lateral accel: 0.371 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.504 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :507 / 510 aka 99.412%

better :486 / 510 aka 95.294%

best (0.5m):271 / 510 aka 53.137%

global mae x: 0.158

global mae y: 0.177

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.186

avg resets: 0.016

Mean abs curvature: 0.050 deg/m

Mean abs curvature rate: 0.093 deg/ms

Mean abs lateral accel: 0.178 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.311 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :297 / 347 aka 85.591%

better :178 / 347 aka 51.297%

best (0.5m):67 / 347 aka 19.308%

global mae x: 0.307

global mae y: 0.273

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.274

avg resets: 0.493

Mean abs curvature: 0.438 deg/m

Mean abs curvature rate: 0.636 deg/ms

Mean abs lateral accel: 0.313 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.442 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :918 / 938 aka 97.868%

better :812 / 938 aka 86.567%

best (0.5m):378 / 938 aka 40.299%

global mae x: 0.178

global mae y: 0.199

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.195

avg resets: 0.071

Mean abs curvature: 0.123 deg/m

Mean abs curvature rate: 0.168 deg/ms

Mean abs lateral accel: 0.321 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.392 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :123 / 133 aka 92.481%

better :70 / 133 aka 52.632%

best (0.5m):12 / 133 aka 9.023%

global mae x: 0.251

global mae y: 0.261

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.236

avg resets: 0.203

Mean abs curvature: 0.298 deg/m

Mean abs curvature rate: 0.340 deg/ms

Mean abs lateral accel: 0.663 m/(ss)

Mean abs lateral accel GT: 0.646 m/(ss)

Mean abs lateral jerk: 0.621 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

32/120 aka 26.67%

better: 10/120 aka 8.33%

best (0.5m): 2/120 aka 1.67%

mean max error C3: 2.144840

mean max error C2: 2.411136

global mae: 0.328650

avg resets: 0.458333

Right lane change segments

40/150 aka 26.67%

better: 12/150 aka 8.00%

best (0.5m): 2/150 aka 1.33%

mean max error C3: 2.344678

mean max error C2: 2.451838

global mae: 0.360372

avg resets: 0.660000

Left turn segments

68/79 aka 86.08%

better: 20/79 aka 25.32%

best (0.5m): 1/79 aka 1.27%

mean max error C3: 1.323250

mean max error C2: 1.400233

global mae: 0.293974

avg resets: 0.873418

Right turn segments

77/91 aka 84.62%

better: 27/91 aka 29.67%

best (0.5m): 3/91 aka 3.30%

mean max error C3: 1.322312

mean max error C2: 1.382699

global mae: 0.291609

avg resets: 1.021978

Left keep segments

22/33 aka 66.67%

better: 13/33 aka 39.39%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.484547

mean max error C2: 1.691937

global mae: 0.280160

avg resets: 0.424242

Right keep segments

19/26 aka 73.08%

better: 7/26 aka 26.92%

best (0.5m): 1/26 aka 3.85%

mean max error C3: 2.024102

mean max error C2: 1.428444

global mae: 0.299489

avg resets: 0.230769

Plan y error in turns (> 6°/s)

Mean path error : -0.027 meters

Mean absolute path error : 0.443 meters

Turn bias test (y_pos at 50m > 3m)

got 29069 left samples

got 30815 right samples

mean left gt: -0.013105523

mean right gt: 0.018673118

mean turn lanecenter error (GT pos, GT ll): left = 0.21726641058921814, right = -0.24762772023677826

mean turn lanecenter error (sim pos, GT ll): left = 0.22112822532653809, right = -0.25129204988479614

mean turn lanecenter error (GT pos, model ll): left = 0.18402014672756195, right = -0.23940737545490265

mean turn lanecenter error (sim pos, model ll): left = 0.16712908446788788, right = -0.20201589167118073

Mean abs curvature: 0.214595 deg/m

Mean abs curvature rate: 0.214599 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.175704 Hz, period 5.691399 s

Mean amplitude: 0.137552 m

Mean ping-pong significance (frequency x amplitude): 2.336235 %