Model analysis report for

▶
root: {} 11 keys
- report_name: 'mlsimdriving_driving.analyse_lat_mlsim'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- shuffle: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_lat_mlsim'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-harald_2025-08-15_16-18-00'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 32 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
      - ▶
        ml_model_config: {} 8 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: '3bec45a4-9512-4c7f-bdbc-9415e5890fb6/100'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-harald_2025-08-15_16-18-00/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 11 keys
    - supercombo: 'a8f96b93-bde2-4e28-a732-4df21ebba968/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1362 / 1588 aka 85.768%

better :979 / 1588 aka 61.650%

best (0.5m):295 / 1588 aka 18.577%

global mae x: 0.171

global mae y: 0.286

global mae y of lanecenter (when detected) in GT: 0.210

global mae y of lanecenter (when detected) in Sim: 0.251

avg resets: 0.256

Mean abs curvature: 0.195 deg/m

Mean abs curvature rate: 0.245 deg/ms

Mean abs lateral accel: 0.360 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.399 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :483 / 491 aka 98.371%

better :451 / 491 aka 91.853%

best (0.5m):204 / 491 aka 41.548%

global mae x: 0.105

global mae y: 0.211

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.204

avg resets: 0.022

Mean abs curvature: 0.040 deg/m

Mean abs curvature rate: 0.072 deg/ms

Mean abs lateral accel: 0.172 m/(ss)

Mean abs lateral accel GT: 0.129 m/(ss)

Mean abs lateral jerk: 0.279 m/(sss)

Mean abs lateral jerk GT: 0.170 m/(sss)

Low speed

Metrics

passed :264 / 334 aka 79.042%

better :164 / 334 aka 49.102%

best (0.5m):58 / 334 aka 17.365%

global mae x: 0.239

global mae y: 0.323

global mae y of lanecenter (when detected) in GT: 0.270

global mae y of lanecenter (when detected) in Sim: 0.272

avg resets: 0.410

Mean abs curvature: 0.381 deg/m

Mean abs curvature rate: 0.488 deg/ms

Mean abs lateral accel: 0.309 m/(ss)

Mean abs lateral accel GT: 0.306 m/(ss)

Mean abs lateral jerk: 0.349 m/(sss)

Mean abs lateral jerk GT: 0.227 m/(sss)

No desire segments

Metrics

passed :845 / 900 aka 93.889%

better :721 / 900 aka 80.111%

best (0.5m):268 / 900 aka 29.778%

global mae x: 0.128

global mae y: 0.240

global mae y of lanecenter (when detected) in GT: 0.181

global mae y of lanecenter (when detected) in Sim: 0.220

avg resets: 0.088

Mean abs curvature: 0.106 deg/m

Mean abs curvature rate: 0.129 deg/ms

Mean abs lateral accel: 0.313 m/(ss)

Mean abs lateral accel GT: 0.282 m/(ss)

Mean abs lateral jerk: 0.342 m/(sss)

Mean abs lateral jerk GT: 0.235 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :105 / 125 aka 84.000%

better :60 / 125 aka 48.000%

best (0.5m):8 / 125 aka 6.400%

global mae x: 0.220

global mae y: 0.304

global mae y of lanecenter (when detected) in GT: 0.263

global mae y of lanecenter (when detected) in Sim: 0.272

avg resets: 0.232

Mean abs curvature: 0.265 deg/m

Mean abs curvature rate: 0.253 deg/ms

Mean abs lateral accel: 0.656 m/(ss)

Mean abs lateral accel GT: 0.652 m/(ss)

Mean abs lateral jerk: 0.515 m/(sss)

Mean abs lateral jerk GT: 0.397 m/(sss)

Left lane change segments

104/116 aka 89.66%

better: 68/116 aka 58.62%

best (0.5m): 10/116 aka 8.62%

mean max error C3: 1.041073

mean max error C2: 1.183538

global mae: 0.290274

avg resets: 0.077586

Right lane change segments

124/147 aka 84.35%

better: 74/147 aka 50.34%

best (0.5m): 14/147 aka 9.52%

mean max error C3: 1.105700

mean max error C2: 1.256425

global mae: 0.298246

avg resets: 0.183673

Left turn segments

50/75 aka 66.67%

better: 18/75 aka 24.00%

best (0.5m): 0/75 aka 0.00%

mean max error C3: 1.732796

mean max error C2: 1.746264

global mae: 0.365192

avg resets: 0.760000

Right turn segments

67/85 aka 78.82%

better: 28/85 aka 32.94%

best (0.5m): 0/85 aka 0.00%

mean max error C3: 1.431692

mean max error C2: 1.513572

global mae: 0.379650

avg resets: 0.729412

Left keep segments

24/30 aka 80.00%

better: 15/30 aka 50.00%

best (0.5m): 0/30 aka 0.00%

mean max error C3: 1.356581

mean max error C2: 1.280352

global mae: 0.296737

avg resets: 0.200000

Right keep segments

18/25 aka 72.00%

better: 11/25 aka 44.00%

best (0.5m): 1/25 aka 4.00%

mean max error C3: 1.388636

mean max error C2: 1.376196

global mae: 0.334694

avg resets: 0.240000

Plan y error in turns (> 6°/s)

Mean path error : -0.020 meters

Mean absolute path error : 0.537 meters

Turn bias test (y_pos at 50m > 3m)

got 27712 left samples

got 29927 right samples

mean left gt: -0.051698558

mean right gt: -0.0074761286

mean turn lanecenter error (GT pos, GT ll): left = 0.22312705218791962, right = -0.24712879955768585

mean turn lanecenter error (sim pos, GT ll): left = 0.1884998381137848, right = -0.24908854067325592

mean turn lanecenter error (GT pos, model ll): left = 0.22274592518806458, right = -0.23831138014793396

mean turn lanecenter error (sim pos, model ll): left = 0.14950984716415405, right = -0.208038330078125

Mean abs curvature: 0.211311 deg/m

Mean abs curvature rate: 0.160257 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.166435 Hz, period 6.008346 s

Mean amplitude: 0.143556 m

Mean ping-pong significance (frequency x amplitude): 2.286058 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	160
turn_right	170
lane_change_left	223
lane_change_right	270
keep_left	76
keep_right	65

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.261m
turn_right	+5.550m
lane_change_left	-1.425m
lane_change_right	+1.338m
keep_left	-1.021m
keep_right	+1.083m

Mean desire duration - ground truth

Command	Duration
turn_left	6.816s
turn_right	6.161s
lane_change_left	3.305s
lane_change_right	3.505s
keep_left	3.763s
keep_right	3.862s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.39s	3.14s	3.84s	4.49s
turn_right	2.25s	3.03s	3.62s	4.33s
lane_change_left	1.24s	1.73s	2.46s	3.99s
lane_change_right	1.23s	1.79s	2.56s	4.06s
keep_left	1.50s	2.41s	3.30s	4.32s
keep_right	1.87s	2.90s	3.90s	5.60s

Group	Gas prob	Brake prob	Steer prob
All segments	0.011	0.023	0.276
Easy straight	0.007	0.009	0.154
Low speed	0.020	0.036	0.380
No desire segments	0.009	0.014	0.217
No desire turns (max lat accel > 2.5 m/ss)	0.013	0.028	0.373