Model analysis report for

▶
root: {} 11 keys
- report_name: 'mlsimdriving_driving.analyse_lat_mlsim'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- shuffle: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_lat_mlsim'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-harald_2025-07-25_08-15-05'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 32 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
      - ▶
        ml_model_config: {} 8 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: '3bec45a4-9512-4c7f-bdbc-9415e5890fb6/76'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-harald_2025-07-25_08-15-05/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 11 keys
    - supercombo: 'f35eaf21-ef51-413d-9857-398bcdabb7da/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1440 / 1587 aka 90.737%

better :1061 / 1587 aka 66.856%

best (0.5m):370 / 1587 aka 23.314%

global mae x: 0.166

global mae y: 0.252

global mae y of lanecenter (when detected) in GT: 0.210

global mae y of lanecenter (when detected) in Sim: 0.232

avg resets: 0.269

Mean abs curvature: 0.210 deg/m

Mean abs curvature rate: 0.341 deg/ms

Mean abs lateral accel: 0.372 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.505 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :486 / 491 aka 98.982%

better :460 / 491 aka 93.686%

best (0.5m):239 / 491 aka 48.676%

global mae x: 0.104

global mae y: 0.190

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.191

avg resets: 0.018

Mean abs curvature: 0.043 deg/m

Mean abs curvature rate: 0.087 deg/ms

Mean abs lateral accel: 0.181 m/(ss)

Mean abs lateral accel GT: 0.129 m/(ss)

Mean abs lateral jerk: 0.339 m/(sss)

Mean abs lateral jerk GT: 0.170 m/(sss)

Low speed

Metrics

passed :294 / 333 aka 88.288%

better :198 / 333 aka 59.459%

best (0.5m):74 / 333 aka 22.222%

global mae x: 0.233

global mae y: 0.278

global mae y of lanecenter (when detected) in GT: 0.269

global mae y of lanecenter (when detected) in Sim: 0.262

avg resets: 0.444

Mean abs curvature: 0.417 deg/m

Mean abs curvature rate: 0.691 deg/ms

Mean abs lateral accel: 0.320 m/(ss)

Mean abs lateral accel GT: 0.306 m/(ss)

Mean abs lateral jerk: 0.462 m/(sss)

Mean abs lateral jerk GT: 0.227 m/(sss)

No desire segments

Metrics

passed :866 / 899 aka 96.329%

better :761 / 899 aka 84.650%

best (0.5m):323 / 899 aka 35.929%

global mae x: 0.124

global mae y: 0.213

global mae y of lanecenter (when detected) in GT: 0.181

global mae y of lanecenter (when detected) in Sim: 0.204

avg resets: 0.082

Mean abs curvature: 0.112 deg/m

Mean abs curvature rate: 0.172 deg/ms

Mean abs lateral accel: 0.322 m/(ss)

Mean abs lateral accel GT: 0.282 m/(ss)

Mean abs lateral jerk: 0.420 m/(sss)

Mean abs lateral jerk GT: 0.235 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :108 / 125 aka 86.400%

better :65 / 125 aka 52.000%

best (0.5m):5 / 125 aka 4.000%

global mae x: 0.213

global mae y: 0.284

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.259

avg resets: 0.248

Mean abs curvature: 0.272 deg/m

Mean abs curvature rate: 0.349 deg/ms

Mean abs lateral accel: 0.667 m/(ss)

Mean abs lateral accel GT: 0.651 m/(ss)

Mean abs lateral jerk: 0.651 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

105/116 aka 90.52%

better: 78/116 aka 67.24%

best (0.5m): 17/116 aka 14.66%

mean max error C3: 0.881959

mean max error C2: 1.077240

global mae: 0.244854

avg resets: 0.077586

Right lane change segments

129/147 aka 87.76%

better: 80/147 aka 54.42%

best (0.5m): 17/147 aka 11.56%

mean max error C3: 0.988330

mean max error C2: 1.183595

global mae: 0.273870

avg resets: 0.170068

Left turn segments

57/75 aka 76.00%

better: 21/75 aka 28.00%

best (0.5m): 0/75 aka 0.00%

mean max error C3: 1.413204

mean max error C2: 1.682444

global mae: 0.313580

avg resets: 0.906667

Right turn segments

72/85 aka 84.71%

better: 37/85 aka 43.53%

best (0.5m): 4/85 aka 4.71%

mean max error C3: 1.160318

mean max error C2: 1.385225

global mae: 0.322259

avg resets: 0.811765

Left keep segments

27/30 aka 90.00%

better: 13/30 aka 43.33%

best (0.5m): 2/30 aka 6.67%

mean max error C3: 1.315919

mean max error C2: 1.209402

global mae: 0.292801

avg resets: 0.233333

Right keep segments

20/25 aka 80.00%

better: 10/25 aka 40.00%

best (0.5m): 3/25 aka 12.00%

mean max error C3: 1.223744

mean max error C2: 1.275961

global mae: 0.294720

avg resets: 0.200000

Plan y error in turns (> 6°/s)

Mean path error : -0.033 meters

Mean absolute path error : 0.466 meters

Turn bias test (y_pos at 50m > 3m)

got 27712 left samples

got 29927 right samples

mean left gt: -0.0018932351

mean right gt: 0.011186138

mean turn lanecenter error (GT pos, GT ll): left = 0.22312705218791962, right = -0.24712882936000824

mean turn lanecenter error (sim pos, GT ll): left = 0.23471476137638092, right = -0.2317577451467514

mean turn lanecenter error (GT pos, model ll): left = 0.2272397130727768, right = -0.2309781163930893

mean turn lanecenter error (sim pos, model ll): left = 0.21398818492889404, right = -0.18633806705474854

Mean abs curvature: 0.212325 deg/m

Mean abs curvature rate: 0.190291 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.174509 Hz, period 5.730367 s

Mean amplitude: 0.132048 m

Mean ping-pong significance (frequency x amplitude): 2.180717 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	160
turn_right	170
lane_change_left	223
lane_change_right	270
keep_left	76
keep_right	65

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-6.013m
turn_right	+5.546m
lane_change_left	-1.492m
lane_change_right	+1.360m
keep_left	-1.265m
keep_right	+0.982m

Mean desire duration - ground truth

Command	Duration
turn_left	6.816s
turn_right	6.161s
lane_change_left	3.305s
lane_change_right	3.505s
keep_left	3.763s
keep_right	3.862s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.38s	3.12s	3.76s	4.28s
turn_right	2.39s	2.99s	3.58s	4.30s
lane_change_left	1.10s	1.54s	2.14s	3.64s
lane_change_right	1.12s	1.65s	2.37s	3.92s
keep_left	1.55s	2.14s	2.85s	4.52s
keep_right	1.66s	2.53s	3.77s	5.36s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.276
Easy straight	0.007	0.009	0.153
Low speed	0.017	0.037	0.380
No desire segments	0.008	0.014	0.216
No desire turns (max lat accel > 2.5 m/ss)	0.012	0.028	0.370