Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable two D457s #144

Merged
merged 22 commits into from
Jan 31, 2023
Merged

Enable two D457s #144

merged 22 commits into from
Jan 31, 2023

Conversation

jnzw
Copy link
Contributor

@jnzw jnzw commented Dec 16, 2022

  • Supports max 4 simultaneous streams from two cameras with limitations
  • Use Nvidia SerDes API to set up max9295 and max9296
  • Add 2nd D457's 4 new sensor nodes, 2nd serializer, gmsl-links in dts
  • Configure SerDes video pipes at run time
    * Fix Y12I resolution bug (1600x1300 -> 1280x800)
  • Build single .deb package
pipe 1st D457 2nd D457
X Depth IR
Y RGB IMU
Z IR Depth
U IMU RGB

(Depth xor IR,RGB xor IMU)

@Nir-Az Nir-Az requested a review from dmipx December 16, 2022 09:57
@xzhangxa xzhangxa requested a review from ev-mp December 17, 2022 12:58
@xzhangxa
Copy link
Collaborator

The title should be "enable two D457s on one max9296" which could be more precise to what the series are doing. Potentially these changes could support multiple max9296s with each has 2 D457 connected, we've kept that in mind but because the HW setup we can't really test it.

It's kept in mind that one D457 should work the same as before. When using two D457 on both link A/B of one max9296, because of the max9296 limitation of 4 streams, it's not possible to stream all streams of 2 D457s but the current setup is fine to make the most useful 2depth+2color and 2ir+2color work.

There's a known issue to be fixed later: DFU functionality only work for Link B D457 now.

@jnzw
Copy link
Contributor Author

jnzw commented Dec 22, 2022

There's a known issue to be fixed later: DFU functionality only work for Link B D457 now.

The issue has been fixed. It was mainly a fw (firmware update counter) and hardware issue, not driver related.

@ev-mp
Copy link
Collaborator

ev-mp commented Dec 22, 2022

@xzhangxa , I'm not sure I understand - each camera can generate up to two streams simultaneously (and use two lanes) ?
What about running 4 streams from camera 1 and zero from camera 2 ?
Can you explain the architecture behind it?

  • how the DTS is configured
  • is it backward-compatible with single camera configuration?
  • What happens if you try to start 2nd camera when the required pipe is in use by the 1st camera?
  • What about metadata, is it available for both cameras?

@xzhangxa
Copy link
Collaborator

@xzhangxa , I'm not sure I understand - each camera can generate up to two streams simultaneously (and use two lanes) ? What about running 4 streams from camera 1 and zero from camera 2 ? Can you explain the architecture behind it?

  • how the DTS is configured
  • is it backward-compatible with single camera configuration?
  • What happens if you try to start 2nd camera when the required pipe is in use by the 1st camera?
  • What about metadata, is it available for both cameras?

Hi @ev-mp

The quick answer is that these changes will not change the single D457 functionality, when 2 D457s are connected on both link A/B input of max9296, each D457 can still be used as before: all 4 streams streaming together, and metadata is still available.

And it's possible to use 1-4 streams from both D457s. Though because of max9295/9296 4 pipe limitation, 8 streams of 2 D457s have to share the 4 pipe. As in the table, when one stream uses a pipe the other D457's corresponding stream will not be able to run (user will get an error). The stream pairs are chosen so that the most common dual-D457 usage 2depth+2color, 2ir+2color are possible.

If necessary @jnzw and I could give a short introduction about all the changes, just summarize more than the PR description.

@jnzw
Copy link
Contributor Author

jnzw commented Jan 13, 2023

Rebased onto latest Jetpack-5.0.2 branch.

@jnzw
Copy link
Contributor Author

jnzw commented Jan 20, 2023

Rebased again onto Jetpack-5.0.2 branch.

@xzhangxa
Copy link
Collaborator

This PR includes the 2 commits from #137 , as there are some comments in that PR and the code was changed a little and @jnzw is on vacation, the maintainer/viewer please ignore that 2 commits and drop them when merging. Thanks.

@jnzw
Copy link
Contributor Author

jnzw commented Jan 24, 2023

@Nir-Az Sorry I'm on vacation and just saw the messages this morning. I have just rebased this PR on top of #137 to include @xzhangxa 's recent changes to the build shell scripts and README. It should now be ready to be merged.

@Nir-Az
Copy link
Collaborator

Nir-Az commented Jan 24, 2023

@Nir-Az Sorry I'm on vacation and just saw the messages this morning. I have just rebased this PR on top of #137 to include @xzhangxa 's recent changes to the build shell scripts and README. It should now be ready to be merged.

Thanks! We will check.

@dmipx
Copy link
Contributor

dmipx commented Jan 26, 2023

This should be rebased as we merged deb PR's

xzhangxa and others added 12 commits January 30, 2023 11:31
Signed-off-by: Xin Zhang <xin.x.zhang@intel.com>
Signed-off-by: Xin Zhang <xin.x.zhang@intel.com>
* Set up GMSL links and init max9295 and max9296 settings from d4xx
* Add proxy ser device and gmsl-link in DT

Signed-off-by: Junze Wu <junze.wu@intel.com>
Calling the GMSL APIs only on the first sensor based on the device tree
link to max9295, so for a typical 2 D457 on link A/B to the same
max9296, these initial setting will be executed twice like the reference
drivers.

This should be considered as a temporary solution. Merging 4 sensors
into one D457 device or changing GMSL API should be better.

Signed-off-by: Xin Zhang <xin.x.zhang@intel.com>
Signed-off-by: Junze Wu <junze.wu@intel.com>
Signed-off-by: Junze Wu <junze.wu@intel.com>
* Disable Depth, IR, IMU sensors of 1st D457 in DT
* Enable second serializer (ser_b) in DT
* Change dser's max-src from 1 to 2
* Add second D457 RGB sensor with metadata enabled

Signed-off-by: Junze Wu <junze.wu@intel.com>
This patch is part of the original PR:
IntelRealSense/realsense_mipi_platform_driver#107

Only the left cleanup is moved here, the IR metadata change in that PR
is not included.

Signed-off-by: Xin Zhang <xin.x.zhang@intel.com>
* Modify pipe Z to pull Depth & EMB
* Modify pipe U to pull RGB & EMB
* Modify DT to add two depth sensors

Signed-off-by: Junze Wu <junze.wu@intel.com>
Signed-off-by: Kaili Xu <kaili.xu@intel.com>
1. Remove hardcode register settings for each pipeline, and change to
use dynamic load pipe config base on the info read from device tree.

2. Change API max9295/9296_update_pipe() to max9295/9296_set_pipe(), and
remove setting pipe related registers in max9295/9296_init_setting(), do it in
max9295/9296_set_pipe() base on the info read from device tree.

3. Remove update pipe for IR pipe in tegra channel setting, and do it in d4xx
start streaming stage. In addition, not limited to the IR pipe type, all types
of pipe setting are performed here.

4. Remove param: "ir_type","ir_type_value", and probe/init check from max9295/9296,
since all the pipe setting flow and needed info are maintained in d4xx.

5. Split 0x0312 from map_pipe_opt in max9296, and set this register for each pipe,
in order to support dynamic switching of pipelines developed in the next step.

6. Add max9295/9296_reset_control() in ds5_remove().

Signed-off-by: Kaili Xu <kaili.xu@intel.com>
Signed-off-by: Xin Zhang <xin.x.zhang@intel.com>
Signed-off-by: Junze Wu <junze.wu@intel.com>
* Add max9296_power_on/off to d4xx
* Fix max9295_reset_control to reset all only if ref count is 0

Signed-off-by: Junze Wu <junze.wu@intel.com>
jnzw and others added 10 commits January 30, 2023 11:31
* Increase num-channels to 8
* Add sensor nodes with gmsl-link sections
* Add vi and nvcsi nodes
* Remove source node from DT
* vc-id is set as follows as a workaround for conflict
| 0 | 1 | 2 | 3 |
| depth0 | color0 | ir0 | imu0 |
| ir1 | imu1 | depth1 | color1 |

Signed-off-by: Junze Wu <junze.wu@intel.com>
Signed-off-by: Xin Zhang <xin.x.zhang@intel.com>
* Acquire available pipe id when streaming starts
* Release video pipe when streaming is done
* Allow streams with same vc-ids to share a pipe

Signed-off-by: Junze Wu <junze.wu@intel.com>
Default to y for setup with maxim serdes.

Signed-off-by: Xin Zhang <xin.x.zhang@intel.com>
Signed-off-by: Kaili Xu <kaili.xu@intel.com>
* Update SerDes pipe setup
* Add API for max9296 one-shot reset
* Do one-shot reset only when start/stop Y12I stream

Signed-off-by: Junze Wu <junze.wu@intel.com>
Signed-off-by: Junze Wu <junze.wu@intel.com>
- Remove some serdes address settings since the original 9295/9296 code
  doesn't set them, and removing them doesn't impact D457 usage.
- Still need to set pipes at init time to make it work, but reuse existing
  code to reduce duplicated lines.
- Use existing address macros instead of hardcode values.

Signed-off-by: Xin Zhang <xin.x.zhang@intel.com>
* Create new DT without serializer B and the 4 nodes of 2nd camera
* Update shell scripts to apply correct DT
* Updated README

Signed-off-by: Junze Wu <junze.wu@intel.com>
Signed-off-by: Junze Wu <junze.wu@intel.com>
Signed-off-by: Junze Wu <junze.wu@intel.com>
@Nir-Az Nir-Az changed the base branch from Jetpack-5.0.2 to jp5-2d457-1-serdes January 31, 2023 12:11
@Nir-Az Nir-Az changed the base branch from jp5-2d457-1-serdes to Jetpack-5.0.2 January 31, 2023 12:12
@Nir-Az Nir-Az changed the base branch from Jetpack-5.0.2 to jp5-2d457-1-des January 31, 2023 12:23
@Nir-Az Nir-Az changed the base branch from jp5-2d457-1-des to Jetpack-5.0.2 January 31, 2023 12:25
@Nir-Az Nir-Az changed the base branch from Jetpack-5.0.2 to jp5-2d457-1-des January 31, 2023 12:25
@Nir-Az Nir-Az merged commit 283037f into IntelRealSense:jp5-2d457-1-des Jan 31, 2023
@jnzw jnzw mentioned this pull request Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants