Model analysis report for

▶
root: {} 13 keys
- report_name: 'mlsimdriving_driving.analyse_disagreements'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- shuffle: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_disagreements'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: '/code.nfs/xx'
  - force_local: false
- codedir2: '/code.nfs/xx'
- ▶
  rollout: {} 10 keys
  - type: 'RawVideoRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 22 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.rawvideo_env.RawVideoEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: true
    - fps: 20
    - allow_lat: true
    - allow_long: true
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
      - ▶
        ml_model_config: {} 8 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: '3bec45a4-9512-4c7f-bdbc-9415e5890fb6/100'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-harald_2025-08-15_16-18-00/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
    - with_gt: false
  - ▶
    agent: {} 11 keys
    - supercombo: 'a8f96b93-bde2-4e28-a732-4df21ebba968/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys
- ▶
  rollout2: {} 10 keys
  - type: 'RawVideoRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 22 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.rawvideo_env.RawVideoEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: true
    - fps: 20
    - allow_lat: true
    - allow_long: true
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
      - ▶
        ml_model_config: {} 8 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: '3bec45a4-9512-4c7f-bdbc-9415e5890fb6/100'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-harald_2025-08-15_16-18-00/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
    - with_gt: false
  - ▶
    agent: {} 11 keys
    - supercombo: 'default_supercombo'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

total 1588 segments

Easy straights lane positioning (>20mph)

- 40 chunks (8 seconds each)

--- (GT) avg = 0.152(m) from lane center

--- (a8f96b93) avg = -0.0000053(1/m) desired curvature

--- (default_supercombo) avg = -0.0000475(1/m) desired curvature

examples with (a8f96b93) more steer:

7bc25afb4957a716|2021-02-24--05-46-23/18

436363c17591bdd7|2021-04-18--14-09-27/30

examples with (default_supercombo) more steer:

1a074a6025c49094|2022-08-23--16-07-51/15

63c37b400c6929dc|2024-05-16--14-34-42/18

bf0fac55013ab937|2021-12-09--10-55-41/49

\

Turns lane positioning

- 38 chunks (8 seconds each)

--- (GT) avg = 0.424(m) from lane center

--- (a8f96b93) avg = 0.0019438(1/m) desired curvature

--- (default_supercombo) avg = 0.0019625(1/m) desired curvature

examples with (a8f96b93) more steer:

f2802051ee0638cf|2021-07-01--19-11-16/146

cc3de51b7ebd943d|2023-12-30--17-13-48/32

examples with (default_supercombo) more steer:

b2e9858e29db492b|2021-08-14--14-05-17/14

a382c127b282777d|2021-04-24--13-03-14/184

2fdb3b00e818dfd7|2023-04-18--10-16-04/47

\

Highway straights speed convergence (50-80mph)

- 27 chunks (8 seconds each)

--- (a8f96b93) avg = -0.2068598(m/s^2) vs GT

--- (default_supercombo) avg = 0.0076525(m/s^2) vs GT

examples with (a8f96b93) faster:

538ff5efc4f13c44|2020-12-03--07-46-54/15

8e3d043fc5c63bc2|2020-12-17--16-02-11/13

bfa5404195bad1a2|2021-07-16--11-21-31/17

52b08febc15038a5|2020-12-01--21-10-11/42

examples with (default_supercombo) faster:

1a074a6025c49094|2022-08-23--16-07-51/15

7bc25afb4957a716|2021-02-24--05-46-23/18

436363c17591bdd7|2021-04-18--14-09-27/30

92e13d6c269f512d|2023-02-12--17-01-29/185

436363c17591bdd7|2021-06-05--05-58-24/41

\

City straights speed convergence (20-40mph)

- 30 chunks (8 seconds each)

--- (a8f96b93) avg = 0.0311112(m/s^2) vs GT

--- (default_supercombo) avg = -0.0667434(m/s^2) vs GT

examples with (a8f96b93) faster:

436363c17591bdd7|2021-08-04--13-47-53/13

63c37b400c6929dc|2024-05-16--14-34-42/18

436363c17591bdd7|2021-08-04--13-47-53/13

examples with (default_supercombo) faster:

5e3a19ff9d0eb905|2021-03-03--11-02-56/23

27858928f280bac1|2021-07-08--22-13-34/19

b76770e1128e5f73|2021-11-18--11-18-29/16

\

Slowdown for turns (decel performance)

- 51 chunks (8 seconds each)

--- (a8f96b93) avg = 0.0260252(m/s^2) vs GT

--- (default_supercombo) avg = -0.0739146(m/s^2) vs GT

examples with (a8f96b93) less brake:

0cf857b1b13bd7cd|2024-01-04--19-27-35/13

a382c127b282777d|2021-04-24--13-03-14/184

0cf857b1b13bd7cd|2024-01-04--19-27-35/13

examples with (default_supercombo) less brake:

5594273f79bc0abe|2023-10-14--12-37-33/17

405e0805a6deab95|2023-04-13--20-07-20/24

\

Come to stop (decel performance)

- 108 chunks (8 seconds each)

--- (a8f96b93) avg = 0.1203107(m/s^2) vs GT

--- (default_supercombo) avg = 0.0922722(m/s^2) vs GT

examples with (a8f96b93) less brake:

aebcd28a78188aa0|2021-07-01--18-03-20/21

d200d0706179dc01|2021-11-16--06-25-13/20

examples with (default_supercombo) less brake:

796d06594c1ffd40|2021-05-15--10-11-58/37

\

Start from stop (accel performance)

- 121 chunks (8 seconds each)

--- (a8f96b93) avg = 0.0643456(m/s^2) vs GT

--- (default_supercombo) avg = -0.0355324(m/s^2) vs GT

examples with (a8f96b93) more accel:

ee44ae76e783280f|2021-12-10--07-10-18/14

examples with (default_supercombo) more accel:

4cef06d33690da48|2021-07-01--11-41-38/13

796d06594c1ffd40|2021-05-15--10-11-58/37

\