Model analysis report for

▶
root: {} 13 keys
- report_name: 'mlsimdriving_driving.analyse_disagreements'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- shuffle: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_disagreements'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: '/code.nfs/xx'
  - force_local: false
- codedir2: '/code.nfs/xx'
- ▶
  rollout: {} 10 keys
  - type: 'RawVideoRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 22 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.rawvideo_env.RawVideoEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: true
    - fps: 20
    - allow_lat: true
    - allow_long: true
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
      - ▶
        ml_model_config: {} 8 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: '3bec45a4-9512-4c7f-bdbc-9415e5890fb6/76'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-harald_2025-07-25_08-15-05/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
    - with_gt: false
  - ▶
    agent: {} 11 keys
    - supercombo: 'f35eaf21-ef51-413d-9857-398bcdabb7da/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys
- ▶
  rollout2: {} 10 keys
  - type: 'RawVideoRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 22 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.rawvideo_env.RawVideoEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: true
    - fps: 20
    - allow_lat: true
    - allow_long: true
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
      - ▶
        ml_model_config: {} 8 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: '3bec45a4-9512-4c7f-bdbc-9415e5890fb6/76'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-harald_2025-07-25_08-15-05/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
    - with_gt: false
  - ▶
    agent: {} 11 keys
    - supercombo: 'default_supercombo'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

total 1644 segments

Easy straights lane positioning (>20mph)

- 43 chunks (8 seconds each)

--- (GT) avg = 0.152(m) from lane center

--- (f35eaf21) avg = -0.0000329(1/m) desired curvature

--- (default_supercombo) avg = -0.0000599(1/m) desired curvature

examples with (f35eaf21) more steer:

7bc25afb4957a716|2021-02-24--05-46-23/18

examples with (default_supercombo) more steer:

1a074a6025c49094|2022-08-23--16-07-51/15

\

Turns lane positioning

- 40 chunks (8 seconds each)

--- (GT) avg = 0.407(m) from lane center

--- (f35eaf21) avg = 0.0017051(1/m) desired curvature

--- (default_supercombo) avg = 0.0017691(1/m) desired curvature

examples with (f35eaf21) more steer:

cc3de51b7ebd943d|2023-12-30--17-13-48/32

f2802051ee0638cf|2021-07-01--19-11-16/146

examples with (default_supercombo) more steer:

ebce2b7ad530a9de|2021-09-28--08-47-49/31

b2e9858e29db492b|2021-08-14--14-05-17/14

e87c38b2bd0844ca|2021-03-06--06-26-21/49

\

Highway straights speed convergence (50-80mph)

- 27 chunks (8 seconds each)

--- (f35eaf21) avg = 0.2582913(m/s^2) vs GT

--- (default_supercombo) avg = 0.0076073(m/s^2) vs GT

examples with (f35eaf21) faster:

7bc25afb4957a716|2021-02-24--05-46-23/18

538ff5efc4f13c44|2020-12-03--07-46-54/15

2ee538cb37aed733|2020-12-26--07-54-13/50

3cadb0e566518d81|2020-12-24--18-12-32/95

8e3d043fc5c63bc2|2020-12-17--16-02-11/13

examples with (default_supercombo) faster:

84fa6d922979ab2f|2024-05-01--15-05-02/16

1a074a6025c49094|2022-08-23--16-07-51/15

92e13d6c269f512d|2023-02-12--17-01-29/185

\

City straights speed convergence (20-40mph)

- 31 chunks (8 seconds each)

--- (f35eaf21) avg = -0.1642396(m/s^2) vs GT

--- (default_supercombo) avg = -0.0620135(m/s^2) vs GT

examples with (f35eaf21) faster:

436363c17591bdd7|2021-07-23--16-25-43/17

436363c17591bdd7|2021-08-04--13-47-53/13

5e3a19ff9d0eb905|2021-03-03--11-02-56/23

examples with (default_supercombo) faster:

365c75cab101a4e1|2021-01-06--16-22-51/20

365c75cab101a4e1|2021-01-06--16-22-51/20

\

Slowdown for turns (decel performance)

- 55 chunks (8 seconds each)

--- (f35eaf21) avg = 0.1469985(m/s^2) vs GT

--- (default_supercombo) avg = -0.0810961(m/s^2) vs GT

examples with (f35eaf21) less brake:

0cf857b1b13bd7cd|2024-01-04--19-27-35/13

a10480a74559b82c|2021-01-04--06-58-27/14

examples with (default_supercombo) less brake:

5594273f79bc0abe|2023-10-14--12-37-33/17

f32fe8f12e987785|2020-11-30--16-37-46/15

5405cac0c9d58c3d|2023-11-04--23-01-08/10

\

Come to stop (decel performance)

- 112 chunks (8 seconds each)

--- (f35eaf21) avg = 0.1453236(m/s^2) vs GT

--- (default_supercombo) avg = 0.0951119(m/s^2) vs GT

examples with (f35eaf21) less brake:

d200d0706179dc01|2021-11-16--06-25-13/20

041a7c231874241f|2021-07-18--04-13-21/11

examples with (default_supercombo) less brake:

81c6e6b0b352c419|2021-03-12--10-10-22/32

\

Start from stop (accel performance)

- 122 chunks (8 seconds each)

--- (f35eaf21) avg = -0.0724729(m/s^2) vs GT

--- (default_supercombo) avg = -0.0372227(m/s^2) vs GT

examples with (f35eaf21) more accel:

725f0f2582cc6fda|2021-10-14--13-23-54/40

a4153d4a8fa7cfee|2021-02-12--18-49-51/22

examples with (default_supercombo) more accel:

876bd1d4539784b0|2021-01-20--11-39-22/14

\