A problem when applying Filter and Align together

[tarekmuallim]

Required Info
Camera Model	D400
Operating System & Version	Win (10)
Platform	PC
SDK Version	SDK 2.0 Build 2.10.4

I am trying to: get a frameset, then apply spatial filter and temporal filter, after that do alignment between the color and the depth.
But my program crash such as "AccessViolationException".

This is my code:

frames = pipe.wait_for_frames();						
framesf = frames; // filtered framesetes
framesf = spat_filter.process(framesf);//apply filter
framesf = temp_filter.process(framesf);//apply filter
aligned_frames = align.process(framesf);
depth = aligned_frames.get_depth_frame();
points = pc.calculate(depth);

The program crash when it reach calculate

I tried to do alignment first then do filtering but also it crash when I apply the filter

I think the Filter and Align can't use in the same time.
How can I fix this problem?

[dorodnic]

Hi @tarekmuallim
This should work, we will try to reproduce and see what happened.
Did you try reversing the order (first align, then post-processing)?
Also, what language do you use?

[tarekmuallim]

Hi @dorodnic
Thanks for the replay.
I am using C++ on Visual Studio 2017 and Windows 10.
Windows SDK version: 10.0.16299.0
Platform toolset: Visual Studio 2017 (v141)
Platform: x64

My code is a part of a bigger program. But I tried to make a new simple one for test.
This the full simple program:

#include <librealsense2/rs.hpp> 


rs2::pointcloud pc;
rs2::points points;

rs2::spatial_filter spat_filter;
rs2::temporal_filter temp_filter;

rs2::frameset frames;
rs2::frameset framesf;
rs2::frameset aligned_frames;

rs2::align align(rs2_stream::RS2_STREAM_COLOR);

int main() {

	rs2::pipeline pipe;
	auto profile = pipe.start();

	spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_MAGNITUDE, 5.0f);
	spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_DELTA, 50.0f);
	spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.3f);

	temp_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_DELTA, 100.0f);
	temp_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.3f);

	frames = pipe.wait_for_frames();
	framesf = frames;
	framesf = spat_filter.process(framesf);//apply filter
	framesf = temp_filter.process(framesf);//apply filter
	aligned_frames = align.process(framesf);
	auto depth = aligned_frames.get_depth_frame();
	points = pc.calculate(depth);

}

The program crash when it reach calculate (last statement).
I tried this also:

#include <librealsense2/rs.hpp> 


rs2::pointcloud pc;
rs2::points points;

rs2::spatial_filter spat_filter;
rs2::temporal_filter temp_filter;

rs2::frameset frames;
rs2::frameset framesf;
rs2::frameset aligned_frames;

rs2::align align(rs2_stream::RS2_STREAM_COLOR);

int main() {

	rs2::pipeline pipe;
	auto profile = pipe.start();

	spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_MAGNITUDE, 5.0f);
	spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_DELTA, 50.0f);
	spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.3f);

	temp_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_DELTA, 100.0f);
	temp_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.3f);

	frames = pipe.wait_for_frames();
	aligned_frames = align.process(frames);
	framesf = aligned_frames;
	framesf = spat_filter.process(framesf);//apply filter
	framesf = temp_filter.process(framesf);//apply filter
	auto depth = framesf.get_depth_frame();
	points = pc.calculate(depth);

}

With this program it takes some time then it crashed. Again the program crash when it reach calculate.

[dorodnic]

Hi @tarekmuallim
Please try the following:

#include <librealsense2/rs.hpp> 

rs2::pointcloud pc;
rs2::points points;

rs2::spatial_filter spat_filter;
rs2::temporal_filter temp_filter;

rs2::frameset frames;
rs2::frameset framesf;
rs2::frameset aligned_frames;

rs2::align align(rs2_stream::RS2_STREAM_COLOR);

int main() {
    rs2::pipeline pipe;
    auto profile = pipe.start();

    spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_MAGNITUDE, 5.0f);
    spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_DELTA, 50.0f);
    spat_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.3f);

    temp_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_DELTA, 100.0f);
    temp_filter.set_option(rs2_option::RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.3f);

    frames = pipe.wait_for_frames();
    aligned_frames = align.process(frames);
    
    // auto depth = framesf.get_depth_frame();
    rs2::frame depth;
    for (auto&& f : aligned_frames)
    {
        if (f.get_profile().format() == RS2_FORMAT_Z16) 
            depth = f;
    }

    depth = spat_filter.process(depth);//apply filter
    depth = temp_filter.process(depth);//apply filter
    points = pc.calculate(depth);
}

[dorodnic]

Thank you for insightful and easy to reproduce feedback.

There are several issues highlighted by the two snippets:

1. Currently, post-processing filters need to operate on individual frame basis. It would be nice to extend the code to enable users to apply filters on entire framesets (enhancement) and we are planning to add it.
2. After applying align, the stream_type of all resulting frames becomes the stream you are aligning to. This, in theory, was to distinguish between real depth frame and a synthetic one (generated from other view-port). An unintended side effect of this design decision is that get_depth_frame() can no longer be used to find the depth frame after alignment. We will think how to improve this, as it is very unintuitive.
3. When single frame (result of post-processing) was being passed to align an error was logged, but not thrown, violating projects error handling policy. This is a (bug) we should address.

measure example can also offer some working reference.

[tarekmuallim]

Thank you for the explanation.
I will try your code for now.
Thanks again.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@tarekmuallim
The code snippet provided by dorodnic should fix your problem, also the enhancement code had been merged into our code base. Can I close the issue?

[tarekmuallim]

@RealSense-Customer-Engineering
yes you can.
thanks a lot for the help.