Model analysis report for

▶
root: {} 12 keys
- report_name: 'mlsimdriving_driving.analyse_disagreements'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_disagreements'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: '/code.nfs/xx'
  - force_local: false
- codedir2: '/code.nfs/xx'
- ▶
  rollout: {} 10 keys
  - type: 'RawVideoRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 22 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.rawvideo_env.RawVideoEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: true
    - fps: 20
    - allow_lat: true
    - allow_long: true
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'COROLLA'
      - ▶
        ml_model_config: {} 6 keys
        
        platform: 'COROLLA'
        
        eid: 'd3e5ef06-14ec-4e35-b708-d8db73d29f1e/51'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
    - with_gt: false
  - ▶
    agent: {} 12 keys
    - supercombo: 'fef4d896-597e-4c6f-90cf-8eb7102c89d2/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys
- ▶
  rollout2: {} 10 keys
  - type: 'RawVideoRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 22 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.rawvideo_env.RawVideoEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: true
    - fps: 20
    - allow_lat: true
    - allow_long: true
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 4 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        classic_model_config: {} 1 keys
        
        platform: 'COROLLA'
      - ▶
        ml_model_config: {} 6 keys
        
        platform: 'COROLLA'
        
        eid: 'd3e5ef06-14ec-4e35-b708-d8db73d29f1e/51'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
    - with_gt: false
  - ▶
    agent: {} 12 keys
    - supercombo: 'default_supercombo'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

total 1644 segments

Easy straights lane positioning (>20mph)

- 43 chunks (8 seconds each)

--- (GT) avg = 0.152(m) from lane center

--- (fef4d896) avg = -0.0000424(1/m) desired curvature

--- (default_supercombo) avg = -0.0000599(1/m) desired curvature

examples with (fef4d896) more steer:

ead14b5ba31cdb4d|2023-08-01--18-38-20/104

436363c17591bdd7|2021-06-05--05-58-24/41

d4e14caac2799607|2022-04-15--23-00-56/10

examples with (default_supercombo) more steer:

bf0fac55013ab937|2021-12-09--10-55-41/49

1a074a6025c49094|2022-08-23--16-07-51/15

\

Turns lane positioning

- 40 chunks (8 seconds each)

--- (GT) avg = 0.407(m) from lane center

--- (fef4d896) avg = 0.0016367(1/m) desired curvature

--- (default_supercombo) avg = 0.0017691(1/m) desired curvature

examples with (fef4d896) more steer:

f2802051ee0638cf|2021-07-01--19-11-16/146

examples with (default_supercombo) more steer:

a382c127b282777d|2021-04-24--13-03-14/184

b2e9858e29db492b|2021-08-14--14-05-17/14

ebce2b7ad530a9de|2021-09-28--08-47-49/31

2fdb3b00e818dfd7|2023-04-18--10-16-04/47

\

Highway straights speed convergence (50-80mph)

- 27 chunks (8 seconds each)

--- (fef4d896) avg = -0.5225623(m/s^2) vs GT

--- (default_supercombo) avg = 0.0075936(m/s^2) vs GT

examples with (fef4d896) faster:

7bc25afb4957a716|2021-02-24--05-46-23/18

538ff5efc4f13c44|2020-12-03--07-46-54/15

examples with (default_supercombo) faster:

1a074a6025c49094|2022-08-25--16-42-39/15

1a074a6025c49094|2022-10-15--05-44-42/15

4205b2b0855e235b|2020-04-03--16-41-09/10

8e3d043fc5c63bc2|2020-12-17--16-02-11/13

\

City straights speed convergence (20-40mph)

- 31 chunks (8 seconds each)

--- (fef4d896) avg = -0.3776538(m/s^2) vs GT

--- (default_supercombo) avg = -0.0620500(m/s^2) vs GT

examples with (fef4d896) faster:

1437b815e783f695|2022-03-18--18-21-37/22

436363c17591bdd7|2021-08-04--13-47-53/13

b76770e1128e5f73|2022-03-26--12-40-45/12

examples with (default_supercombo) faster:

9f054450b372d46e|2020-12-19--14-25-52/23

63c37b400c6929dc|2024-05-16--14-34-42/18

\

Slowdown for turns (decel performance)

- 55 chunks (8 seconds each)

--- (fef4d896) avg = -0.4573189(m/s^2) vs GT

--- (default_supercombo) avg = -0.0812054(m/s^2) vs GT

examples with (fef4d896) less brake:

da8e662b3acac579|2021-11-27--12-09-40/11

0cf857b1b13bd7cd|2024-01-04--19-27-35/13

a10480a74559b82c|2021-01-04--06-58-27/14

b794e7af372e5724|2022-05-21--09-57-39/16

0b6e3a5ec849ddf7|2023-12-20--17-26-27/25

examples with (default_supercombo) less brake:

e7bf8f732c0f6b12|2021-06-14--12-46-14/93

cc3de51b7ebd943d|2023-12-30--17-13-48/32

\

Come to stop (decel performance)

- 112 chunks (8 seconds each)

--- (fef4d896) avg = 0.1929661(m/s^2) vs GT

--- (default_supercombo) avg = 0.0951136(m/s^2) vs GT

examples with (fef4d896) less brake:

5a2be05bbbd06fc5|2024-01-24--03-33-31/42

46f4f80fce2f2ac8|2020-12-17--14-10-32/13

8b9bae324529e447|2020-12-19--15-10-36/104

examples with (default_supercombo) less brake:

041a7c231874241f|2021-07-18--04-13-21/11

796d06594c1ffd40|2021-05-15--10-11-58/37

873884f8a54c5558|2021-10-13--21-56-35/20

\

Start from stop (accel performance)

- 122 chunks (8 seconds each)

--- (fef4d896) avg = -0.0504240(m/s^2) vs GT

--- (default_supercombo) avg = -0.0372153(m/s^2) vs GT

examples with (fef4d896) more accel:

ee44ae76e783280f|2021-12-10--07-10-18/14

examples with (default_supercombo) more accel:

6fea1470edf96da9|2024-06-03--19-18-06/31

0d557534de7bc3a8|2024-05-12--11-49-04/11

\