Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rsdk 2077 write gray16 images to viam depth format #1960

Merged
merged 10 commits into from
Mar 3, 2023
Merged

Rsdk 2077 write gray16 images to viam depth format #1960

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3c09325
allow gray16 to be converted to viam custom type
bhaney Mar 1, 2023
86875b2
WIP
bhaney Mar 2, 2023
1c3a503
adds tests for viam custom depth
bhaney Mar 2, 2023
dcabb35
grayscale 16-bit images can also be encoded as raw viam depthmap
bhaney Mar 2, 2023
3e95637
check for lazy image type
bhaney Mar 2, 2023
21a43a7
change doc string
bhaney Mar 2, 2023
1975f42
error string edits
bhaney Mar 2, 2023
0426613
use the DM fake file
bhaney Mar 2, 2023
69a54c8
linting and add more comments
bhaney Mar 2, 2023
e617722
linting and add more comments
bhaney Mar 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .artifact/tree.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3881,8 +3881,8 @@
"size": 567
},
"fakeDM.vnd.viam.dep": {
"hash": "81d039d0786ecda6a9d8f1fb40ff9cba",
"size": 1624
"hash": "4a7dca803d353064c65886fae49c823a",
"size": 424
},
"go_encoded_image.png": {
"hash": "ae17174de7ec900b670aa2230ce83db9",
Expand Down
132 changes: 107 additions & 25 deletions rimage/depth_map_raw.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"bufio"
"compress/gzip"
"encoding/binary"
"image"
"io"
"os"
"path/filepath"
Expand All @@ -18,7 +19,7 @@ import (
const MagicNumIntVersionX = 6363110499870197078

// MagicNumIntViamType is the magic number (as an int) for the custom Viam depth type.
// magic number for ViamCustomType is int64([]byte("DEPTHMAP")).
// magic number for ViamCustomType is uint64([]byte("DEPTHMAP")), Little Endian.
const MagicNumIntViamType = 5782988369567958340

func _readNext(r io.Reader) (int64, error) {
Expand Down Expand Up @@ -61,15 +62,16 @@ func ReadDepthMap(r io.Reader) (*DepthMap, error) {
}
switch firstBytes {
case MagicNumIntVersionX: // magic number for VERSIONX
return readDepthMapVersionX(r.(*bufio.Reader))
return readDepthMapVersionX(r)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is already a great change just from synchronizing the function signatures to all input a io.Reader. Love this.

case MagicNumIntViamType: // magic number for ViamCustomType is int64([]byte("DEPTHMAP"))
return readDepthMapViam(r.(*bufio.Reader))
return readDepthMapViam(r)
default:
return readDepthMapRaw(r.(*bufio.Reader), firstBytes)
return readDepthMapRaw(r, firstBytes)
}
}

func readDepthMapRaw(f *bufio.Reader, firstBytes int64) (*DepthMap, error) {
func readDepthMapRaw(ff io.Reader, firstBytes int64) (*DepthMap, error) {
f := bufio.NewReader(ff)
dm := DepthMap{}

dm.width = int(firstBytes)
Expand All @@ -83,26 +85,34 @@ func readDepthMapRaw(f *bufio.Reader, firstBytes int64) (*DepthMap, error) {
return setRawDepthMapValues(f, &dm)
}

func readDepthMapViam(f *bufio.Reader) (*DepthMap, error) {
dm := DepthMap{}
func readDepthMapViam(ff io.Reader) (*DepthMap, error) {
f := bufio.NewReader(ff)
dm := &DepthMap{}

rawWidth, err := _readNext(f)
if err != nil {
return nil, err
return nil, errors.Wrapf(err, "could not read vnd.viam.dep width")
}
dm.width = int(rawWidth)
rawHeight, err := _readNext(f)
if err != nil {
return nil, err
return nil, errors.Wrapf(err, "could not read vnd.viam.dep height")
}
dm.height = int(rawHeight)

return setRawDepthMapValues(f, &dm)
// dump the rest of the bytes in a depth slice
datSlice := make([]Depth, dm.height*dm.width)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a smoother way to do setRawDepthMapValues() except now there's two places with basically the same code chunk. We only use this here and in readDepthMapRaw, so maybe just go ahead and replace the function with this logic? Might mean that the error printout gets slightly less specific though...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like we're just dumping it all out as opposed to reading 8 bytes at a time. Nice. This should also be valid for raw depth maps.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raw Depth maps are actually treated differently:

In the raw depth map function, it is writing the two bytes (uint16) of depth info from the *DepthMap to 8 bytes (uint64) of space in the buffer. These leaves a lot of 0-padding in the files. When reading the Raw Depth file using setRawDepthMapValues, it has to read 8 byte chunks at a time, just to read the 2 bytes of depth to put into the *DepthMap pixel.

This new Viam function, on the other hand, writes out 2 bytes, and reads 2 bytes. So the space the file takes up is smaller, because it doesn't have 6 extra bytes of 0s for every 2 bytes of depth info. This also allows you to just dump the data from the file into a depth slice, without having to parse 8 bytes at a time.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll put a comment in the file pointing out the difference

err = binary.Read(f, binary.LittleEndian, &datSlice)
if err != nil {
return nil, errors.Wrapf(err, "could not read vnd.viam.dep data slice")
}
dm.data = datSlice
return dm, nil
}

func readDepthMapVersionX(r *bufio.Reader) (*DepthMap, error) {
func readDepthMapVersionX(rr io.Reader) (*DepthMap, error) {
dm := DepthMap{}

r := bufio.NewReader(rr)
// get past garbade
_, err := r.ReadString('\n')
if err != nil {
Expand Down Expand Up @@ -183,6 +193,7 @@ func readDepthMapVersionX(r *bufio.Reader) (*DepthMap, error) {
return &dm, nil
}

// setRawDepthMapValues read out values 8 bytes at a time, converting the 8 bytes into a 2 byte depth value.
func setRawDepthMapValues(f *bufio.Reader, dm *DepthMap) (*DepthMap, error) {
if dm.width <= 0 || dm.width >= 100000 || dm.height <= 0 || dm.height >= 100000 {
return nil, errors.Errorf("bad width or height for depth map %v %v", dm.width, dm.height)
Expand All @@ -204,7 +215,7 @@ func setRawDepthMapValues(f *bufio.Reader, dm *DepthMap) (*DepthMap, error) {
}

// WriteRawDepthMapToFile writes the raw depth map to the given file.
func WriteRawDepthMapToFile(dm *DepthMap, fn string) (err error) {
func WriteRawDepthMapToFile(dm image.Image, fn string) (err error) {
//nolint:gosec
f, err := os.Create(fn)
if err != nil {
Expand All @@ -226,15 +237,15 @@ func WriteRawDepthMapToFile(dm *DepthMap, fn string) (err error) {
}

if strings.HasSuffix(fn, ".vnd.viam.dep") {
_, err := out.Write(DepthMapMagicNumber)
_, err = WriteViamDepthMapTo(dm, out)
if err != nil {
return err
}
} else {
_, err = WriteRawDepthMapTo(dm, out)
if err != nil {
return err
}
}

_, err = WriteRawDepthMapTo(dm, out)
if err != nil {
return err
}

if gout != nil {
Expand All @@ -247,27 +258,37 @@ func WriteRawDepthMapToFile(dm *DepthMap, fn string) (err error) {
}

// WriteRawDepthMapTo writes this depth map to the given writer.
func WriteRawDepthMapTo(dm *DepthMap, out io.Writer) (int64, error) {
// the raw depth map type writes 8 bytes of width, 8 bytes of height, and 8 bytes per pixel.
func WriteRawDepthMapTo(img image.Image, out io.Writer) (int64, error) {
buf := make([]byte, 8)
var totalN int64
width := img.Bounds().Dx()
height := img.Bounds().Dy()

binary.LittleEndian.PutUint64(buf, uint64(dm.width))
binary.LittleEndian.PutUint64(buf, uint64(width))
n, err := out.Write(buf)
totalN += int64(n)
if err != nil {
return totalN, err
}

binary.LittleEndian.PutUint64(buf, uint64(dm.height))
binary.LittleEndian.PutUint64(buf, uint64(height))
n, err = out.Write(buf)
totalN += int64(n)
if err != nil {
return totalN, err
}

for x := 0; x < dm.width; x++ {
for y := 0; y < dm.height; y++ {
binary.LittleEndian.PutUint64(buf, uint64(dm.GetDepth(x, y)))
for x := 0; x < width; x++ {
for y := 0; y < height; y++ {
switch dm := img.(type) {
case *DepthMap:
binary.LittleEndian.PutUint64(buf, uint64(dm.GetDepth(x, y)))
case *image.Gray16:
binary.LittleEndian.PutUint64(buf, uint64(dm.Gray16At(x, y).Y))
default:
return totalN, errors.Errorf("cannot convert image type %T to a raw depth format", dm)
}
n, err = out.Write(buf)
totalN += int64(n)
if err != nil {
Expand All @@ -278,3 +299,64 @@ func WriteRawDepthMapTo(dm *DepthMap, out io.Writer) (int64, error) {

return totalN, nil
}

// WriteViamDepthMapTo writes depth map or gray16 image to the given writer as vnd.viam.dep bytes.
// the Viam custom depth type writes 8 bytes of "magic number", 8 bytes of width, 8 bytes of height, and 2 bytes per pixel.
func WriteViamDepthMapTo(img image.Image, out io.Writer) (int64, error) {
if lazy, ok := img.(*LazyEncodedImage); ok {
lazy.decode()
if lazy.decodeErr != nil {
return 0, errors.Errorf("could not decode LazyEncodedImage to a depth image: %v", lazy.decodeErr)
}
img = lazy.decodedImage
}
buf := make([]byte, 8)
var totalN int64
width := img.Bounds().Dx()
height := img.Bounds().Dy()

binary.LittleEndian.PutUint64(buf, uint64(MagicNumIntViamType))
n, err := out.Write(buf)
totalN += int64(n)
if err != nil {
return totalN, err
}
binary.LittleEndian.PutUint64(buf, uint64(width))
n, err = out.Write(buf)
totalN += int64(n)
if err != nil {
return totalN, err
}

binary.LittleEndian.PutUint64(buf, uint64(height))
n, err = out.Write(buf)
totalN += int64(n)
if err != nil {
return totalN, err
}
switch dm := img.(type) {
case *DepthMap:
err = binary.Write(out, binary.LittleEndian, dm.data) // uint16 data
if err != nil {
return totalN, err
}
totalN += int64(len(dm.data) * 2)
case *image.Gray16:
grayBuf := make([]byte, 2)
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
i := dm.PixOffset(x, y)
z := uint16(dm.Pix[i+0])<<8 | uint16(dm.Pix[i+1])
binary.LittleEndian.PutUint16(grayBuf, z)
n, err = out.Write(grayBuf)
totalN += int64(n)
if err != nil {
return totalN, err
}
}
}
default:
return totalN, errors.Errorf("cannot convert image type %T to image/vnd.viam.dep depth format", dm)
}
return totalN, nil
}
38 changes: 38 additions & 0 deletions rimage/depth_map_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,44 @@ func TestDepthColorModel(t *testing.T) {
test.That(t, convGray.(color.Gray16).Y, test.ShouldEqual, 6168)
}

func TestViamDepthMap(t *testing.T) {
// create various types of depth representations
width := 10
height := 20
dm := NewEmptyDepthMap(width, height)
g16 := image.NewGray16(image.Rect(0, 0, width, height))
g8 := image.NewGray(image.Rect(0, 0, width, height))
for x := 0; x < width; x++ {
for y := 0; y < height; y++ {
dm.Set(x, y, Depth(x*y))
g16.SetGray16(x, y, color.Gray16{uint16(x * y)})
g8.SetGray(x, y, color.Gray{uint8(x * y)})
}
}
// write dm to a viam type buffer
buf := &bytes.Buffer{}
byt, err := WriteViamDepthMapTo(dm, buf)
test.That(t, err, test.ShouldBeNil)
test.That(t, byt, test.ShouldEqual, (3*8 + 2*width*height)) // 3 bytes for header, 2 bytes per pixel
// write gray16 to a viam type buffer
buf16 := &bytes.Buffer{}
byt16, err := WriteViamDepthMapTo(g16, buf16)
test.That(t, err, test.ShouldBeNil)
test.That(t, byt16, test.ShouldEqual, (3*8 + 2*width*height)) // 3 bytes for header, 2 bytes per pixel
// gray should fail
buf8 := &bytes.Buffer{}
_, err = WriteViamDepthMapTo(g8, buf8)
test.That(t, err, test.ShouldNotBeNil)
test.That(t, err.Error(), test.ShouldContainSubstring, "cannot convert image type")
// read from a viam type buffers and compare to original
dm2, err := ReadDepthMap(buf)
test.That(t, err, test.ShouldBeNil)
test.That(t, dm2, test.ShouldResemble, dm)
dm3, err := ReadDepthMap(buf16)
test.That(t, err, test.ShouldBeNil)
test.That(t, dm3, test.ShouldResemble, dm)
}

func TestDepthMapEncoding(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we lost the "make sure you can read this from a file with the right extension" test and we might wanna keep it. The depth map in artifact/rimage/fakeDM.vnd.viam.dep is just like the one created below from scratch.

Copy link
Member Author

@bhaney bhaney Mar 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR changes the way vnd.viam.dep files are written. Each value is not given 8 bytes of space in the file, but only two bytes. This "compression" makes the new file sizes smaller, but also means that the format is different from what it was before.

But yes, I will create a new vnd.viam.dep file without the 8byte padding, and just the 2byte per pixel length.

Copy link
Member Author

@bhaney bhaney Mar 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

0000000   D   E   P   T   H   M   A   P 
0000010  024  \0  \0  \0  \0  \0  \0  \0
0000020  \n  \0  \0  \0  \0  \0  \0  \0  
0000030  \0  \0  \0  \0  \0  \0  \0  \0
0000040  \0  \0  \0  \0  \0  \0  \0  \0  
0000050  \0  \0  \0  \0  \0  \0  \0  \0
0000060  \0  \0  \0  \0  \0  \0  \0  \0  
0000070  \0  \0  \0  \0  \0  \0  \0  \0
0000100  \0  \0  \0  \0  \0  \0  \0  \0  
0000110  \0  \0  \0  \0  \0  \0  \0  \0
0000120  \0  \0  \0  \0  \0  \0  \0  \0  
0000130  \0  \0  \0  \0  \0  \0  \0  \0
0000140  \0  \0  \0  \0  \0  \0  \0  \0  
0000150  \0  \0  \0  \0  \0  \0  \0  \0
0000160  001  \0  \0  \0  \0  \0  \0  \0 
0000170  002  \0  \0  \0  \0  \0  \0  \0
0000200  003  \0  \0  \0  \0  \0  \0  \0
0000210  004 \0  \0  \0  \0  \0  \0  \0

Like, here is the byte writeout for artifact/rimage/fakeDM.vnd.viam.dep, where each line is 8 bytes, and the first 3 lines are the DEPTHMAP header, width and height respectively. And then after that, each 8 bytes is one pixel

Copy link
Member Author

@bhaney bhaney Mar 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here is the new vnd.viam.dep format, which is more compact, which essentially holds 4 pixels worth of data on each line

0000000   D   E   P   T   H   M   A   P 
0000010  024  \0  \0  \0  \0  \0  \0  \0
0000020  \n  \0  \0  \0  \0  \0  \0  \0  
0000030  \0  \0  \0  \0  \0  \0  \0  \0
0000040  \0  \0  \0  \0  \0  \0  \0  \0  
0000050  \0  \0  \0  \0  \0  \0  \0  \0
0000060  \0  \0  \0  \0  \0  \0  \0  \0  
0000070  \0  \0  \0  \0  \0  \0  \0  \0
0000100  \0  \0 001  \0 002  \0 003  \0 
0000110  004  \0 005  \0 006  \0  \a  \0
0000120  \b  \0  \t  \0  \n  \0  \v  \0  
0000130  \f  \0  \r  \0 016  \0 017  \0
0000140  020  \0 021  \0 022  \0 023  \0  
0000150  \0  \0 002  \0 004  \0 006  \0

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced the fakeDM file in artifact to new more compressed format.

m, err := NewDepthMapFromFile(context.Background(), artifact.MustPath("rimage/fakeDM.vnd.viam.dep"))
test.That(t, err, test.ShouldBeNil)
Expand Down
7 changes: 2 additions & 5 deletions rimage/image_file.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,7 @@ func init() {
// image.Decode as long as we have the appropriate header
image.RegisterFormat("vnd.viam.dep", string(DepthMapMagicNumber),
func(r io.Reader) (image.Image, error) {
f := r.(*bufio.Reader)
dm, err := ReadDepthMap(f)
dm, err := ReadDepthMap(r)
if err != nil {
return nil, err
}
Expand Down Expand Up @@ -293,9 +292,7 @@ func EncodeImage(ctx context.Context, img image.Image, mimeType string) ([]byte,
bounds := img.Bounds()
switch actualOutMIME {
case ut.MimeTypeRawDepth:
buf.Write(DepthMapMagicNumber)
// WriteRawDepthMapTo encodes the height and width
if _, err := WriteRawDepthMapTo(img.(*DepthMap), &buf); err != nil {
if _, err := WriteViamDepthMapTo(img, &buf); err != nil {
return nil, err
}
case ut.MimeTypeRawRGBA:
Expand Down