Computer Vision for TMQ global orientation: considerations for and possible implementation #20
Assignees
Labels
dev:enhancement
New feature or request
Comments
|
@zepumph helped me set up tracking using MarkerInput.js and it worked really well out of the box. It seems like orientation information for a marker is readily available and the data comes in quick. I could easily see adding support for 1) with this method. I imagine 2) could be done as well, but maybe it will present challenges like the ones mentioned in phetsims/tangible#7 |
|
|
Performance concerns to be investigated before/during implementation (marker detection, not sim performance):
|
|
Looping in @emily-phet as an FYI ahead of meeting on Tuesday |
|
Regarding Color tracking: JG:
[Brett Fiedler] [Jesse Greenberg] |
|
We'll move forward with OpenCV for marker tracking. Beholder is not intended for robust motion tracking (deblurring). @jessegreenberg will implement and we will figure out how far we can get with single marker (global rotation) and multiple marker tracking (vertex tracking) in the context of the quadrilateral |
|
I got a pretty consistent (much better than #20 (comment)) angle tracking working by watching two green rectangles, finding the centers of their contours, and then determining the angle of the line between them. This gets around the issue of not knowing the relative orientation of a single rectangle (which could go back to zero degrees every 90 degrees). The green works better than red to pick filter out in the image.
|
|
I connected the above to the sim, its not too bad at all!
EDIT: test code for this: <!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>OPENCV TEST</title>
</head>
<script async src="opencv.js" onload="startOpenCv()" type="text/javascript"></script>
<body>
<video id="video-element"></video>
<canvas id="transfer-canvas"></canvas>
<!--<canvas id="output-canvas"></canvas>-->
<!--<canvas id="contour-canvas"></canvas>-->
<iframe id="sim-frame"
src="quadrilateral_en_phet.html?deviceConnection&postMessageOnLoad"
width="800" height="600"></iframe>
<div>
<label id="min-h-label" for="min-h">Min h:</label>
<input type="range" id="min-h" min="0" max="255" value="0">
<label id="min-s-label" for="min-s">Min s:</label>
<input type="range" id="min-s" min="0" max="255" value="0">
<label id="min-v-label" for="min-v">Min v:</label>
<input type="range" id="min-v" min="0" max="255" value="0">
<label id="max-h-label" for="max-h">max h:</label>
<input type="range" id="max-h" min="0" max="255" value="150">
<label id="max-s-label" for="max-s">max s:</label>
<input type="range" id="max-s" min="0" max="255" value="150">
<label id="max-v-label" for="max-v">max v:</label>
<input type="range" id="max-v" min="0" max="255" value="150">
</div>
</body>
<script>
// connect to simulation
let simulationModel = null;
const iframe = document.getElementById( 'sim-frame' );
window.addEventListener( 'message', event => {
if ( !event.data ) {
return;
}
let data;
try {
data = JSON.parse( event.data );
}
catch( e ) {
return;
}
if ( data.type === 'load' ) {
simulationModel = iframe.contentWindow.simModel;
console.log( simulationModel );
}
} );
class Side {
constructor( p1, p2 ) {
this.p1 = p1;
this.p2 = p2;
this.length = Math.sqrt( ( p2.x - p1.x ) * ( p2.x - p1.x ) + ( p2.y - p1.y ) * ( p2.y - p1.y ) );
this.angle = Math.atan2( p2.y - p1.y, p2.x - p1.x ) + Math.PI / 2;
}
}
const sliderIds = [
"min-h",
"min-s",
"min-v",
"max-h",
"max-s",
"max-v"
];
sliderIds.forEach( sliderId => {
const slider = document.getElementById( sliderId );
slider.addEventListener( 'input', () => {
document.getElementById( `${sliderId}-label` ).textContent = `${sliderId}: ${slider.value}`
} );
} )
// these values work OK to detect the green on my quad
// min: [63, 46, 87, 0]
// max: [91, 150, 246, 255]
const getMinFilterValues = () => {
return [
parseInt( document.getElementById( 'min-h' ).value, 10 ),
parseInt( document.getElementById( 'min-s' ).value, 10 ),
parseInt( document.getElementById( 'min-v' ).value, 10 ),
0 // alpha
]
}
const getMaxFilterValues = () => {
return [
parseInt( document.getElementById( 'max-h' ).value, 10 ),
parseInt( document.getElementById( 'max-s' ).value, 10 ),
parseInt( document.getElementById( 'max-v' ).value, 10 ),
255 // alpha
]
}
const startOpenCv = () => {
const width = 640;
const height = 480;
// Set up video capture with a webcam
let video = document.getElementById( "video-element" );
navigator.mediaDevices.getUserMedia( { video: true, audio: false } )
.then( function( stream ) {
video.srcObject = stream;
video.play();
} )
.catch( function( err ) {
console.log( "An error occurred! " + err );
} );
let transferCanvas = document.getElementById( "transfer-canvas" );
transferCanvas.width = width;
transferCanvas.height = height;
let context = transferCanvas.getContext( "2d" );
let imageCopyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let filterOutputMat = new cv.Mat( height, width, cv.CV_8UC4 );
let leftRedOutputFilterMat = new cv.Mat( height, width, cv.CV_8UC4 );
let rightRedOutputFilterMat = new cv.Mat( height, width, cv.CV_8UC4 );
let fullRedMaskMat = new cv.Mat( height, width, cv.CV_8UC4 );
let redMaskOutput = new cv.Mat( height, width, cv.CV_8UC4 );
let morphologyOutput = new cv.Mat( height, width, cv.CV_8UC4 );
let onesMat = cv.Mat.ones( 3, 3, cv.CV_8U );
let anchor = new cv.Point( -1, -1 );
let greyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let cannyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let dst = new cv.Mat( height, width, cv.CV_8UC1 );
let contoursDestinationMat = new cv.Mat( height, width, cv.CV_8UC1 );
const FPS = 30;
let color = new cv.Scalar( 255, 0, 0 );
let contoursColor = new cv.Scalar( 255, 255, 255 );
let rectangleColor = new cv.Scalar( 255, 0, 0 );
function processVideo() {
let begin = Date.now();
// get the video data and draw to a temp canvas so that it can be read by cv functions
context.drawImage( video, 0, 0, transferCanvas.width, transferCanvas.height );
imageCopyMat.data.set( context.getImageData( 0, 0, width, height ).data );
let lines = new cv.Mat();
// input canvas
let imageSource = cv.imread( 'transfer-canvas' );
let hsvSource = new cv.Mat();
cv.cvtColor( imageSource, hsvSource, cv.COLOR_RGB2HSV );
// FOr a color that is not red this generally works. These higher and lower values match the green
// of some ducktape I bought.
const lower = [ 44, 94, 90, 255 ];
const higher = [ 88, 250, 239, 255 ];
// use these instead to filter out values by slider
// const lower = getMinFilterValues();
// const higher = getMaxFilterValues();
let lowMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), lower );
let highMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), higher );
// filter out to only detect light green of the hardware green
cv.inRange( hsvSource, lowMat, highMat, filterOutputMat );
//cv.imshow( 'output-canvas', filterOutputMat );
// // for red, the color wraps around 180 in hsv space so we need two filers, from 0 to 10 and 160 to 180
// // left/right as in the left and right ends of h specturm in hsv
// const lowerRedLeft = [ 0, 100, 20, 0 ];
// const upperRedLeft = [ 5, 255, 255, 255 ];
// let lowerRedLeftMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), lowerRedLeft );
// let upperRedLeftMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), upperRedLeft );
//
// const lowerRedRight = [ 178, 100, 20, 0 ];
// const upperRedRight = [ 179, 255, 255, 255 ];
// let lowerRedRightMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), lowerRedRight );
// let upperRedRightMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), upperRedRight );
//
// cv.inRange( hsvSource, lowerRedLeftMat, upperRedLeftMat, leftRedOutputFilterMat );
// cv.inRange( hsvSource, lowerRedRightMat, upperRedRightMat, rightRedOutputFilterMat );
//
// // combine masks
// cv.add( leftRedOutputFilterMat, rightRedOutputFilterMat, fullRedMaskMat );
// cv.bitwise_and( hsvSource, hsvSource, redMaskOutput, fullRedMaskMat );
//cv.imshow( 'output-canvas', redMaskOutput );
// cv.cvtColor( redMaskOutput, greyscaleSource, cv.COLOR_RGB2HSV );
// use morphology to filter out noise
// for red
// cv.morphologyEx( redMaskOutput, morphologyOutput, cv.MORPH_OPEN, onesMat, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
// for green
// cv.morphologyEx( filterOutputMat, morphologyOutput, cv.MORPH_OPEN, onesMat, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
// cv.imshow( 'output-canvas', morphologyOutput );
// convert to greyscale - that is just the V channel of hsv
// let channels = new cv.MatVector();
// cv.split( morphologyOutput, channels );
// cv.imshow( 'output-canvas', channels.get( 2 ) );
// find contours in the filtered image
let contours = new cv.MatVector();
let hierarchy = new cv.Mat();
cv.findContours( filterOutputMat, contours, hierarchy, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE );
// You can try more different parameters
if ( contours.size() > 0 ) {
let largestContourArea = 0;
let largestRotatedRect;
let secondLargestContourArea = 0;
let secondLargestRotatedRect;
const markers = [];
// find the largest two contours contour
for ( let i = 0; i < contours.size(); i++ ) {
const rotatedRect = cv.minAreaRect( contours.get( i ) );
const area = rotatedRect.size.width * rotatedRect.size.height;
if ( area > 100 ) {
markers.push( rotatedRect );
}
}
// draw them
markers.forEach( marker => {
let vertices = cv.RotatedRect.points( marker );
for ( let i = 0; i < 4; i++ ) {
cv.line( contoursDestinationMat, vertices[ i ], vertices[ ( i + 1 ) % 4 ], rectangleColor, 2, cv.LINE_AA, 0 );
}
} );
if ( markers.length === 2 ) {
const firstCenter = markers[ 0 ].center;
const secondCenter = markers[ 1 ].center;
// make sure that frame of reference remains the same
const rightCenter = firstCenter.x > secondCenter.x ? firstCenter : secondCenter;
const leftCenter = rightCenter === firstCenter ? secondCenter : firstCenter;
const deltaX = rightCenter.x - leftCenter.x;
const deltaY = rightCenter.y - leftCenter.y;
const angleRadians = Math.atan2( deltaY, deltaX );
if ( simulationModel ) {
const shapeModel = simulationModel.quadrilateralShapeModel;
const topLength = shapeModel.topSide.lengthProperty.value;
const rightLength = shapeModel.rightSide.lengthProperty.value;
const bottomLength = shapeModel.bottomSide.lengthProperty.value;
const leftLength = shapeModel.leftSide.lengthProperty.value;
const p1Angle = shapeModel.vertexA.angleProperty.value;
const p2Angle = shapeModel.vertexB.angleProperty.value;
const p3Angle = shapeModel.vertexC.angleProperty.value;
const p4Angle = shapeModel.vertexD.angleProperty.value;
simulationModel.markerRotationProperty.value = angleRadians;
shapeModel.setPositionsFromLengthsAndAngles( topLength, rightLength, leftLength, p1Angle, p2Angle, p3Angle, p4Angle );
}
console.log( angleRadians );
}
if ( largestContourArea > 100 ) {
// // draw rotatedRect
// let largestVertices = cv.RotatedRect.points( largestRotatedRect );
// for ( let i = 0; i < 4; i++ ) {
// cv.line( contoursDestinationMat, largestVertices[ i ], largestVertices[ ( i + 1 ) % 4 ], rectangleColor, 2, cv.LINE_AA, 0 );
// }
//
// if ( secondLargestContourArea > 100 ) {
//
// // possible with this approach we don't have a second largest rotated rect
// let secondLargestVertices = cv.RotatedRect.points( secondLargestRotatedRect );
// for ( let i = 0; i < 4; i++ ) {
// cv.line( contoursDestinationMat, secondLargestVertices[ i ], secondLargestVertices[ ( i + 1 ) % 4 ], rectangleColor, 2, cv.LINE_AA, 0 );
// }
// }
// const sideA = new Side( vertices[ 0 ], vertices[ 1 ] );
// const sideB = new Side( vertices[ 1 ], vertices[ 2 ] );
// const sideC = new Side( vertices[ 2 ], vertices[ 3 ] );
// const sideD = new Side( vertices[ 3 ], vertices[ 0 ] );
// const sides = [ sideA, sideB, sideC, sideD ];
// let longestSide = null;
// let longestSideLength = 0;
// for ( let i = 0; i < sides.length; i++ ) {
// let side = sides[ i ];
// if ( side.length > longestSideLength ) {
// longestSideLength = side.length;
// longestSide = side;
// }
// }
//
// // now get the angle of the longest side
// console.log( longestSide.angle );
// rect.angle of opencCV goes to zero around 0, 90, 180, 270 and so on, since those rects seem to have no
// rotation. Lets calculate it instead by the slop of the largest length
// console.log( largestRotatedRect.angle );
}
}
//cv.imshow( 'contour-canvas', contoursDestinationMat );
// clear the contour canvas for next time
contoursDestinationMat.delete();
contoursDestinationMat = cv.Mat.zeros( height, width, cv.CV_8UC3 );
// redMaskOutput.delete();
// redMaskOutput = cv.Mat.zeros( height, width, cv.CV_8UC3 );
lowMat.delete();
highMat.delete();
hsvSource.delete();
//channels.delete();
lines.delete();
imageSource.delete();
contours.delete();
hierarchy.delete();
//lowerRedLeftMat.delete();
//upperRedLeftMat.delete();
//lowerRedRightMat.delete();
//upperRedRightMat.delete();
// cv.cvtColor( imageCopyMat, greyMat, cv.COLOR_RGBA2GRAY, 0 );
// cv.Canny( greyMat, cannyMat, 50, 200, 3 );
//cv.HoughLinesP( cannyMat, lines, 1, Math.PI / 180, 30, 0, 0 );
//cv.HoughLines( cannyMat, lines, 1, Math.PI / 180, 1, 0, 0, 0, Math.PI );
// cv.cvtColor( cannyMat, dst, cv.COLOR_RGBA2GRAY );
// cv.imshow( "output-canvas", dst );
// // draw lines
// for ( let i = 0; i < lines.rows; ++i ) {
// let startPoint = new cv.Point( lines.data32S[ i * 4 ], lines.data32S[ i * 4 + 1 ] );
// let endPoint = new cv.Point( lines.data32S[ i * 4 + 2 ], lines.data32S[ i * 4 + 3 ] );
// cv.line( dst, startPoint, endPoint, color );
// }
// cv.imshow( 'output-canvas', dst );
//
// // clear it for next time
// dst = cv.Mat.zeros( width, height, cv.CV_8UC3 );
// schedule next one.
let delay = 1000 / FPS - ( Date.now() - begin );
window.setTimeout( processVideo, delay );
}
// schedule first one.
window.setTimeout( processVideo, 0 );
};
</script>
</html> |
|
Next, we should try tracking four markers that would control four vertex positions defining the quadrilateral. Over slack @BLFiedler suggested that they could be different colors so that we know how to identify them. We could probably get pretty far without distinguishing each marker with coloring, just reassigning the left most and right most vertex to left most and right most markers. Or we could have different sized markers to label them. I also want to try using a "line detection" approach that may work with any kind of hand occlusion. We could detect the lines of the TMQ, extend them all the way to the edge of the image, find line intersection points, and those would be the locations of our vertices. If any portion of a side is visible we will see vertex positions. https://www.geeksforgeeks.org/line-detection-python-opencv-houghline-method EDIT: Here is another document for hough line detection: https://docs.opencv.org/3.4/d3/de6/tutorial_js_houghlines.html |
|
Trying out Hough Line Transform approach: Starting with this image:
Lines like this can be detected:
With this opencv snippet: let src = cv.imread('canvasInput');
let dst = cv.Mat.zeros(src.rows, src.cols, cv.CV_8UC3);
let lines = new cv.Mat();
cv.cvtColor(src, src, cv.COLOR_RGBA2GRAY, 0);
cv.Canny(src, src, 50, 200, 3);
// You can try more different parameters
cv.HoughLines(src, lines, 1, Math.PI / 180,
50, 0, 0, 0, Math.PI);
// draw lines
for (let i = 0; i < lines.rows; ++i) {
let rho = lines.data32F[i * 2];
let theta = lines.data32F[i * 2 + 1];
let a = Math.cos(theta);
let b = Math.sin(theta);
let x0 = a * rho;
let y0 = b * rho;
let startPoint = {x: x0 - 1000 * b, y: y0 + 1000 * a};
let endPoint = {x: x0 + 1000 * b, y: y0 - 1000 * a};
cv.line(dst, startPoint, endPoint, [255, 0, 0, 255]);
}
cv.imshow('canvasOutput', dst);
src.delete(); dst.delete(); lines.delete();An example of how this could work with occlusion. My hands are covering two vertices entirely but it is able to find the sides.
Here I was able to find the intersection points of lines that are not of equivalent slope:
Maybe I can use k-means clustering to find the centers of each vertex, opencv has a function to do so. Or use morphological operations on that image to create blobs and then contours around clusters of points. Or maybe a different averaging solution. I got close with kmeans I think but ran out of time. Here is code with a TODO for next time. <!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>OPENCV TEST</title>
</head>
<script async src="opencv.js" onload="startOpenCv()" type="text/javascript"></script>
<body>
<video id="video-element"></video>
<canvas id="transfer-canvas"></canvas>
<canvas id="output-canvas"></canvas>
<canvas id="contour-canvas"></canvas>
<canvas id="lines-canvas"></canvas>
<canvas id="intersection-points-canvas"></canvas>
<iframe id="sim-frame"
src="quadrilateral_en_phet.html?deviceConnection&postMessageOnLoad"
width="800" height="600"></iframe>
<div>
<label id="min-h-label" for="min-h">Min h:</label>
<input type="range" id="min-h" min="0" max="255" value="0">
<label id="min-s-label" for="min-s">Min s:</label>
<input type="range" id="min-s" min="0" max="255" value="0">
<label id="min-v-label" for="min-v">Min v:</label>
<input type="range" id="min-v" min="0" max="255" value="0">
<label id="max-h-label" for="max-h">max h:</label>
<input type="range" id="max-h" min="0" max="255" value="150">
<label id="max-s-label" for="max-s">max s:</label>
<input type="range" id="max-s" min="0" max="255" value="150">
<label id="max-v-label" for="max-v">max v:</label>
<input type="range" id="max-v" min="0" max="255" value="150">
</div>
</body>
<script>
// connect to simulation
let simulationModel = null;
let dot = null;
const iframe = document.getElementById( 'sim-frame' );
window.addEventListener( 'message', event => {
if ( !event.data ) {
return;
}
let data;
try {
data = JSON.parse( event.data );
}
catch( e ) {
return;
}
if ( data.type === 'load' ) {
simulationModel = iframe.contentWindow.simModel;
dot = iframe.contentWindow.phet.dot;
kite = iframe.contentWindow.phet.kite;
}
} );
class Side {
constructor( p1, p2 ) {
this.p1 = p1;
this.p2 = p2;
this.length = Math.sqrt( ( p2.x - p1.x ) * ( p2.x - p1.x ) + ( p2.y - p1.y ) * ( p2.y - p1.y ) );
this.angle = Math.atan2( p2.y - p1.y, p2.x - p1.x ) + Math.PI / 2;
}
}
class LineWithAngle {
/**
* Construct a line where the angle of the line is already calculated from an opencv operation.
* @param p1
* @param p2
* @param theta
*/
constructor( p1, p2, theta ) {
this.line = new kite.Line( p1, p2 );
this.angle = theta;
}
}
const sliderIds = [
"min-h",
"min-s",
"min-v",
"max-h",
"max-s",
"max-v"
];
sliderIds.forEach( sliderId => {
const slider = document.getElementById( sliderId );
slider.addEventListener( 'input', () => {
document.getElementById( `${sliderId}-label` ).textContent = `${sliderId}: ${slider.value}`
} );
} )
// these values work OK to detect the green on my quad
// min: [63, 46, 87, 0]
// max: [91, 150, 246, 255]
const getMinFilterValues = () => {
return [
parseInt( document.getElementById( 'min-h' ).value, 10 ),
parseInt( document.getElementById( 'min-s' ).value, 10 ),
parseInt( document.getElementById( 'min-v' ).value, 10 ),
0 // alpha
]
}
const getMaxFilterValues = () => {
return [
parseInt( document.getElementById( 'max-h' ).value, 10 ),
parseInt( document.getElementById( 'max-s' ).value, 10 ),
parseInt( document.getElementById( 'max-v' ).value, 10 ),
255 // alpha
]
}
const startOpenCv = () => {
const width = 640;
const height = 480;
// Set up video capture with a webcam
let video = document.getElementById( "video-element" );
navigator.mediaDevices.getUserMedia( { video: true, audio: false } )
.then( function( stream ) {
video.srcObject = stream;
video.play();
} )
.catch( function( err ) {
console.log( "An error occurred! " + err );
} );
let transferCanvas = document.getElementById( "transfer-canvas" );
transferCanvas.width = width;
transferCanvas.height = height;
let context = transferCanvas.getContext( "2d" );
let imageCopyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let filterOutputMat = new cv.Mat( height, width, cv.CV_8UC4 );
let intersectionPointMat = new cv.Mat( height, width, cv.CV_8UC4 );
let kMeansLabelsMat = new cv.Mat();
const clusterCount = 4;
let greyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let cannyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let erosionMat = new cv.Mat( height, width, cv.CV_8UC4 );
let dst = new cv.Mat( height, width, cv.CV_8UC1 );
const FPS = 30;
function processVideo() {
let begin = Date.now();
// now we rely on the dot library to find line intersections
if ( dot && kite ) {
// get the video data and draw to a temp canvas so that it can be read by cv functions
context.drawImage( video, 0, 0, transferCanvas.width, transferCanvas.height );
imageCopyMat.data.set( context.getImageData( 0, 0, width, height ).data );
let lines = new cv.Mat();
// input canvas
let imageSource = cv.imread( 'transfer-canvas' );
let hsvSource = new cv.Mat();
cv.cvtColor( imageSource, hsvSource, cv.COLOR_RGB2HSV );
// FOr a color that is not red this generally works. These higher and lower values match the green
// of some ducktape I bought.
// const lower = [ 44, 94, 90, 255 ];
// const higher = [ 88, 250, 239, 255 ];
// These values worked better at 4:34 pm light in my apartment
const lower = [ 41, 120, 58, 255 ];
const higher = [ 103, 255, 255, 255 ];
// use these instead to filter out values by slider
// const lower = getMinFilterValues();
// const higher = getMaxFilterValues();
let lowMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), lower );
let highMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), higher );
// filter out to only detect light green of the hardware green
cv.inRange( hsvSource, lowMat, highMat, filterOutputMat );
// cv.imshow( 'output-canvas', filterOutputMat );
// Try eroding first to get clearer lines
let M = cv.Mat.ones( 10, 10, cv.CV_8U );
let anchor = new cv.Point( -1, -1 );
cv.erode( filterOutputMat, erosionMat, M, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
cv.imshow( 'output-canvas', erosionMat );
M.delete();
// an edge detection filter first - using the imageCopyMap is important for some reason, and for some reason
// it has to be grayscale
let cvtColorTempMat = cv.imread( 'output-canvas' );
cv.cvtColor( cvtColorTempMat, greyMat, cv.COLOR_RGBA2GRAY, 0 );
cv.Canny( greyMat, cannyMat, 50, 200, 3 );
//cv.imshow( 'output-canvas', cannyMat );
cv.HoughLines( cannyMat, lines, 1, Math.PI / 180, 30, 0, 0, 0, Math.PI );
//cv.HoughLinesP( cannyMat, lines, 1, Math.PI / 180, 10, 0, 0 );
const linesWithAngle = [];
// draw lines - for HoughLines specifically
for ( let i = 0; i < lines.rows; ++i ) {
let rho = lines.data32F[ i * 2 ];
let theta = lines.data32F[ i * 2 + 1 ];
let a = Math.cos( theta );
let b = Math.sin( theta );
let x0 = a * rho;
let y0 = b * rho;
// these lines just extend well beyond the image beyond the image
let startPoint = { x: x0 - 1000 * b, y: y0 + 1000 * a };
let endPoint = { x: x0 + 1000 * b, y: y0 - 1000 * a };
cv.line( dst, startPoint, endPoint, [ 255, 0, 0, 255 ] );
const p1 = new dot.Vector2( x0 - 1000 * b, y0 + 1000 * a );
const p2 = new dot.Vector2( x0 + 1000 * b, y0 - 1000 * a );
const newLine = new LineWithAngle( p1, p2, theta );
linesWithAngle.push( newLine );
}
const intersectionPoints = [];
// expensive operations here at O(n^2)...
for ( let i = 0; i < linesWithAngle.length; i++ ) {
const lineWithAngleA = linesWithAngle[ i ];
// the intersection point for this line, once we find it we will stop looking
for ( let j = 0; j < linesWithAngle.length; j++ ) {
const lineWithAngleB = linesWithAngle[ j ];
// don't look for intersections with self
if ( i !== j ) {
// if the difference in angle between two lines is too small, it is probably a line of the same or
// opposite side, we don't care about these intersections or there cannot be any
const angleDifference = Math.abs( lineWithAngleA.angle - lineWithAngleB.angle )
if ( angleDifference > Math.PI / 4 && angleDifference < 3 * Math.PI / 4 ) {
// an array with a single kite.SegmentIntersection is returned
const intersection = kite.Line.intersect( lineWithAngleA.line, lineWithAngleB.line )[ 0 ];
if ( intersection ) {
const intersectionPoint = intersection.point;
intersectionPoints.push( intersectionPoint );
}
}
}
}
}
// console.log( intersectionPoints.length );
intersectionPoints.forEach( intersectionPoint => {
// can pass a vector2 right to this, yay
cv.circle( intersectionPointMat, intersectionPoint, 3, [ 0, 0, 255, 255 ] );
} );
cv.imshow( 'intersection-points-canvas', intersectionPointMat );
// use k-means algorithm to find the clusters of intersection points
const kMeansCriteria = new cv.TermCriteria( cv.TermCriteria_EPS + cv.TermCriteria_COUNT, 10, 1.0 );
if ( intersectionPoints.length > 0 ) {
const xYArray = intersectionPoints.flatMap( point => [ point.x, point.y ] );
const centers = new cv.Mat( 4, 2, cv.CV_32F );
// TODO: This is running, but I don't understand the centers output
let kMeansPointsMat = cv.matFromArray( xYArray.length, 2, cv.CV_32F, xYArray );
const compactness = cv.kmeans( kMeansPointsMat, clusterCount, kMeansLabelsMat, kMeansCriteria, 1, cv.KMEANS_PP_CENTERS, centers );
// console.log( centers.length );
kMeansPointsMat.delete();
centers.delete();
}
// draw lines - for HougLinesP specifically
// for ( let i = 0; i < lines.rows; ++i ) {
// let startPoint = new cv.Point( lines.data32S[ i * 4 ], lines.data32S[ i * 4 + 1 ] );
// let endPoint = new cv.Point( lines.data32S[ i * 4 + 2 ], lines.data32S[ i * 4 + 3 ] );
// cv.line( dst, startPoint, endPoint, [ 255, 0, 0, 255 ] );
// }
cv.imshow( 'lines-canvas', dst );
// clear lines output for next time
dst.delete();
dst = cv.Mat.zeros( height, width, cv.CV_8UC3 );
// clear circles output for next time
intersectionPointMat.delete();
intersectionPointMat = cv.Mat.zeros( height, width, cv.CV_8UC3 );
lowMat.delete();
highMat.delete();
hsvSource.delete();
lines.delete();
imageSource.delete();
cvtColorTempMat.delete();
}
// schedule next one.
let delay = 1000 / FPS - ( Date.now() - begin );
window.setTimeout( processVideo, delay );
}
// schedule first one.
window.setTimeout( processVideo, 0 );
};
</script>
</html>kmeans seems overly complicated at this point, I am going to turn each of those blobs into a countour and find the center. I tried an "open" operation but it seems to reduce the framerate substantially: cv.morphologyEx( tempMat, newMat, cv.MORPH_OPEN, Ma, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );Instead, I am just going to create large circles at the intersection points so it looks like a big connected blob OK, here it is altogether:
There is a fair amount of jitter because the lines are unstable. I think a lot of it is coming from the canny edge detection that happens first, look at all this noise:
It is coming from noise in the initial color filter.
Hmm, "convex hull" may be what I want to get something more stable. It isn't really any better. I am trying to find a way to get the "skeleton" of the pixels displayed so there is only a single line but I am not having good luck. Ooo, there is a fitLine function...cv.fitLine(cnt, cv.DIST_L2,0,0.01,0.01) approxPolyDP may be what we need:
approxPolyDB might give us an access to straight lines without noise:
let src = cv.imread('canvasInput');
let dst = cv.Mat.zeros(src.rows, src.cols, cv.CV_8UC3);
cv.cvtColor(src, src, cv.COLOR_RGBA2GRAY, 0);
cv.threshold(src, src, 100, 200, cv.THRESH_BINARY);
let contours = new cv.MatVector();
let hierarchy = new cv.Mat();
let poly = new cv.MatVector();
cv.findContours(src, contours, hierarchy, cv.RETR_CCOMP, cv.CHAIN_APPROX_SIMPLE);
// approximates each contour to polygon
for (let i = 0; i < contours.size(); ++i) {
let tmp = new cv.Mat();
let cnt = contours.get(i);
// You can try more different parameters
cv.approxPolyDP(cnt, tmp, 15, true);
poly.push_back(tmp);
cnt.delete(); tmp.delete();
}
// draw contours with random Scalar
for (let i = 0; i < contours.size(); ++i) {
let color = new cv.Scalar(Math.round(Math.random() * 255), Math.round(Math.random() * 255),
Math.round(Math.random() * 255));
cv.drawContours(dst, poly, i, color, 1, 8, hierarchy, 0);
}
cv.imshow('canvasOutput', dst);
src.delete(); dst.delete(); hierarchy.delete(); contours.delete(); poly.delete();Final code before switching to a four marker solution <!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>OPENCV TEST</title>
</head>
<script async src="opencv.js" onload="startOpenCv()" type="text/javascript"></script>
<body>
<video id="video-element"></video>
<canvas id="transfer-canvas"></canvas>
<canvas id="output-canvas"></canvas>
<canvas id="contour-canvas"></canvas>
<canvas id="lines-canvas"></canvas>
<canvas id="intersection-points-canvas"></canvas>
<canvas id="final-vertex-canvas"></canvas>
<iframe id="sim-frame"
src="quadrilateral_en_phet.html?deviceConnection&postMessageOnLoad"
width="800" height="600"></iframe>
<div>
<label id="min-h-label" for="min-h">Min h:</label>
<input type="range" id="min-h" min="0" max="255" value="0">
<label id="min-s-label" for="min-s">Min s:</label>
<input type="range" id="min-s" min="0" max="255" value="0">
<label id="min-v-label" for="min-v">Min v:</label>
<input type="range" id="min-v" min="0" max="255" value="0">
<label id="max-h-label" for="max-h">max h:</label>
<input type="range" id="max-h" min="0" max="255" value="150">
<label id="max-s-label" for="max-s">max s:</label>
<input type="range" id="max-s" min="0" max="255" value="150">
<label id="max-v-label" for="max-v">max v:</label>
<input type="range" id="max-v" min="0" max="255" value="150">
</div>
</body>
<script>
// connect to simulation
let simulationModel = null;
let dot = null;
const iframe = document.getElementById( 'sim-frame' );
window.addEventListener( 'message', event => {
if ( !event.data ) {
return;
}
let data;
try {
data = JSON.parse( event.data );
}
catch( e ) {
return;
}
if ( data.type === 'load' ) {
simulationModel = iframe.contentWindow.simModel;
dot = iframe.contentWindow.phet.dot;
kite = iframe.contentWindow.phet.kite;
}
} );
class Side {
constructor( p1, p2 ) {
this.p1 = p1;
this.p2 = p2;
this.length = Math.sqrt( ( p2.x - p1.x ) * ( p2.x - p1.x ) + ( p2.y - p1.y ) * ( p2.y - p1.y ) );
this.angle = Math.atan2( p2.y - p1.y, p2.x - p1.x ) + Math.PI / 2;
}
}
class LineWithAngle {
/**
* Construct a line where the angle of the line is already calculated from an opencv operation.
* @param p1
* @param p2
* @param theta
*/
constructor( p1, p2, theta ) {
this.line = new kite.Line( p1, p2 );
this.angle = theta;
}
}
const sliderIds = [
"min-h",
"min-s",
"min-v",
"max-h",
"max-s",
"max-v"
];
sliderIds.forEach( sliderId => {
const slider = document.getElementById( sliderId );
slider.addEventListener( 'input', () => {
document.getElementById( `${sliderId}-label` ).textContent = `${sliderId}: ${slider.value}`
} );
} )
// these values work OK to detect the green on my quad
// min: [63, 46, 87, 0]
// max: [91, 150, 246, 255]
const getMinFilterValues = () => {
return [
parseInt( document.getElementById( 'min-h' ).value, 10 ),
parseInt( document.getElementById( 'min-s' ).value, 10 ),
parseInt( document.getElementById( 'min-v' ).value, 10 ),
0 // alpha
]
}
const getMaxFilterValues = () => {
return [
parseInt( document.getElementById( 'max-h' ).value, 10 ),
parseInt( document.getElementById( 'max-s' ).value, 10 ),
parseInt( document.getElementById( 'max-v' ).value, 10 ),
255 // alpha
]
}
const startOpenCv = () => {
const width = 640;
const height = 480;
// Set up video capture with a webcam
let video = document.getElementById( "video-element" );
navigator.mediaDevices.getUserMedia( { video: true, audio: false } )
.then( function( stream ) {
video.srcObject = stream;
video.play();
} )
.catch( function( err ) {
console.log( "An error occurred! " + err );
} );
let transferCanvas = document.getElementById( "transfer-canvas" );
transferCanvas.width = width;
transferCanvas.height = height;
let context = transferCanvas.getContext( "2d" );
let imageCopyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let filterOutputMat = new cv.Mat( height, width, cv.CV_8UC4 );
let intersectionPointMat = new cv.Mat( height, width, cv.CV_8UC4 );
let finalVertexPointMat = new cv.Mat( height, width, cv.CV_8UC4 );
let copyTempMat = new cv.Mat( height, width, cv.CV_8UC4 );
let kMeansLabelsMat = new cv.Mat();
const clusterCount = 4;
let grayForContoursMat = new cv.Mat( height, width, cv.CV_8UC1 );
let contoursDestinationMat = new cv.Mat( height, width, cv.CV_8UC1 );
let greyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let cannyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let erosionMat = new cv.Mat( height, width, cv.CV_8UC4 );
let dst = new cv.Mat( height, width, cv.CV_8UC1 );
const FPS = 30;
function processVideo() {
let begin = Date.now();
// now we rely on the dot library to find line intersections
if ( dot && kite ) {
// get the video data and draw to a temp canvas so that it can be read by cv functions
context.drawImage( video, 0, 0, transferCanvas.width, transferCanvas.height );
imageCopyMat.data.set( context.getImageData( 0, 0, width, height ).data );
let lines = new cv.Mat();
// input canvas
let imageSource = cv.imread( 'transfer-canvas' );
let hsvSource = new cv.Mat();
cv.cvtColor( imageSource, hsvSource, cv.COLOR_RGB2HSV );
// FOr a color that is not red this generally works. These higher and lower values match the green
// of some ducktape I bought.
// const lower = [ 44, 94, 90, 255 ];
// const higher = [ 88, 250, 239, 255 ];
// These values worked better at 4:34 pm light in my apartment
const lower = [ 41, 120, 58, 255 ];
const higher = [ 103, 255, 255, 255 ];
// use these instead to filter out values by slider
// const lower = getMinFilterValues();
// const higher = getMaxFilterValues();
let lowMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), lower );
let highMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), higher );
// filter out to only detect light green of the hardware green
cv.inRange( hsvSource, lowMat, highMat, filterOutputMat );
// Try eroding first to get clearer lines
let M = cv.Mat.ones( 10, 10, cv.CV_8U );
let anchor = new cv.Point( -1, -1 );
cv.erode( filterOutputMat, erosionMat, M, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
cv.dilate( erosionMat, erosionMat, M, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
cv.imshow( 'output-canvas', erosionMat );
M.delete();
let contours = new cv.MatVector();
let hierarchy = new cv.Mat();
let hull = new cv.MatVector();
let tmpContoursMat = cv.imread( 'output-canvas' );
cv.cvtColor( tmpContoursMat, grayForContoursMat, cv.COLOR_RGB2GRAY );
cv.findContours( grayForContoursMat, contours, hierarchy, cv.RETR_CCOMP, cv.CHAIN_APPROX_SIMPLE );
tmpContoursMat.delete();
// approximate each contour to a hull
for ( let i = 0; i < contours.size(); ++i ) {
let tmp = new cv.Mat();
let cnt = contours.get( i );
cv.convexHull( cnt, tmp, false, true );
hull.push_back( tmp );
cnt.delete();
tmp.delete();
}
for ( let i = 0; i < contours.size(); ++i ) {
let colorHull = new cv.Scalar( Math.round( Math.random() * 255 ), Math.round( Math.random() * 255 ),
Math.round( Math.random() * 255 ) );
cv.drawContours( erosionMat, hull, i, colorHull, 1, 8, hierarchy, 0 );
}
//cv.imshow( 'output-canvas', erosionMat );
hierarchy.delete();
hull.delete();
contours.delete();
// if ( contours.size() > 0 ) {
// for ( let i = 0; i < contours.size(); i++ ) {
// const rotatedRect = cv.minAreaRect( contours.get( i ) );
// const area = rotatedRect.size.width * rotatedRect.size.height;
//
// if ( area > 100 ) {
// let vertices = cv.RotatedRect.points( rotatedRect );
// for ( let i = 0; i < 4; i++ ) {
// cv.line( erosionMat, vertices[ i ], vertices[ ( i + 1 ) % 4 ], [ 255, 0, 0, 255 ], 2, cv.LINE_AA, 0 );
// }
// }
// }
// }
// contours.delete();
//cv.imshow( 'output-canvas', erosionMat );
// an edge detection filter first - using the imageCopyMap is important for some reason, and for some reason
// it has to be grayscale
// imread means that the imshow is CRITICAL
let cvtColorTempMat = cv.imread( 'output-canvas' );
// cv.cvtColor( cvtColorTempMat, greyMat, cv.COLOR_RGBA2GRAY, 0 );
// cv.Canny( greyMat, cannyMat, 50, 200, 3 );
//cv.imshow( 'output-canvas', cannyMat );
//cv.imshow( 'output-canvas', filterOutputMat );
// cv.HoughLines( cannyMat, lines, 1, Math.PI / 180, 20, 0, 0, 0, Math.PI );
// cv.HoughLinesP( cannyMat, lines, 1, Math.PI / 180, 2, 0, 0 );
const linesWithAngle = [];
// draw lines - for HoughLines specifically
for ( let i = 0; i < lines.rows; ++i ) {
let rho = lines.data32F[ i * 2 ];
let theta = lines.data32F[ i * 2 + 1 ];
let a = Math.cos( theta );
let b = Math.sin( theta );
let x0 = a * rho;
let y0 = b * rho;
// these lines just extend well beyond the image beyond the image
let startPoint = { x: x0 - 1000 * b, y: y0 + 1000 * a };
let endPoint = { x: x0 + 1000 * b, y: y0 - 1000 * a };
cv.line( dst, startPoint, endPoint, [ 255, 0, 0, 255 ] );
const p1 = new dot.Vector2( x0 - 1000 * b, y0 + 1000 * a );
const p2 = new dot.Vector2( x0 + 1000 * b, y0 - 1000 * a );
const newLine = new LineWithAngle( p1, p2, theta );
linesWithAngle.push( newLine );
}
// find the intersections of lines
// const intersectionPoints = [];
//
// // expensive operations here at O(n^2)...
// for ( let i = 0; i < linesWithAngle.length; i++ ) {
// const lineWithAngleA = linesWithAngle[ i ];
//
// // the intersection point for this line, once we find it we will stop looking
// for ( let j = 0; j < linesWithAngle.length; j++ ) {
// const lineWithAngleB = linesWithAngle[ j ];
//
// // don't look for intersections with self
// if ( i !== j ) {
//
// // if the difference in angle between two lines is too small, it is probably a line of the same or
// // opposite side, we don't care about these intersections or there cannot be any
// const angleDifference = Math.abs( lineWithAngleA.angle - lineWithAngleB.angle )
// if ( angleDifference > Math.PI / 4 && angleDifference < 3 * Math.PI / 4 ) {
//
// // an array with a single kite.SegmentIntersection is returned
// const intersection = kite.Line.intersect( lineWithAngleA.line, lineWithAngleB.line )[ 0 ];
// if ( intersection ) {
// const intersectionPoint = intersection.point;
// intersectionPoints.push( intersectionPoint );
// }
// }
// }
// }
// }
//
// // console.log( intersectionPoints.length );
// intersectionPoints.forEach( intersectionPoint => {
//
// // can pass a vector2 right to this, yay
// // draw them white so we can use this mat directly to find contours
// cv.circle( intersectionPointMat, intersectionPoint, 10, [ 255, 255, 255, 255 ], cv.FILLED );
// } );
// cv.imshow( 'intersection-points-canvas', intersectionPointMat );
//
// // finally find the vertex positions
//
// cv.cvtColor( intersectionPointMat, grayForContoursMat, cv.COLOR_RGB2GRAY );
//
// const markers = [];
// let contours = new cv.MatVector();
// let hierarchy = new cv.Mat();
// cv.findContours( grayForContoursMat, contours, hierarchy, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE );
//
// if ( contours.size() > 0 ) {
// for ( let i = 0; i < contours.size(); i++ ) {
// if ( markers.length < 4 ) {
// const rotatedRect = cv.minAreaRect( contours.get( i ) );
// const area = rotatedRect.size.width * rotatedRect.size.height;
//
// if ( area > 100 ) {
// markers.push( rotatedRect );
// }
// }
// }
// }
//
// markers.forEach( marker => {
// cv.circle( finalVertexPointMat, marker.center, 25, [ 0, 255, 0, 255 ], cv.FILLED );
// } );
// cv.imshow( 'final-vertex-canvas', finalVertexPointMat );
//
// contours.delete();
// hierarchy.delete();
//
// cv.imshow( 'lines-canvas', dst );
// memory cleanup
// clear lines output for next time
dst.delete();
dst = cv.Mat.zeros( height, width, cv.CV_8UC3 );
// clear circles output for next time
intersectionPointMat.delete();
intersectionPointMat = cv.Mat.zeros( height, width, cv.CV_8UC3 );
finalVertexPointMat.delete();
finalVertexPointMat = cv.Mat.zeros( height, width, cv.CV_8UC3 );
lowMat.delete();
highMat.delete();
hsvSource.delete();
lines.delete();
imageSource.delete();
cvtColorTempMat.delete();
}
// schedule next one.
let delay = 1000 / FPS - ( Date.now() - begin );
window.setTimeout( processVideo, delay );
}
// schedule first one.
window.setTimeout( processVideo, 0 );
};
</script>
</html> |
|
Discussed with @BLFiedler at a check-in meeting. We like the idea of line tracking but lets put that on hold for now.
EDIT: I would like to first play with marker size to accomplish this because it would be easiest. Keeping in mind I think we can pretty quickly change things to support just about anything listed here. The idea is that we could have markers of varying length. Then the height of each marker could still be used to determine perspective if we want. |
|
Notes as I work on a solution that uses 4 discrete markers. Overall, there is hardly any noise and it feels really fast. But it is of course more susceptible to marker occlusion. I made substantial progress on this today, here is my hacky code: <!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>OPENCV TEST</title>
<style>
.hidden {
position: absolute;
height: 100px;
width: 100px;
right: -50px;
top: 50px;
}
</style>
</head>
<script async src="opencv.js" onload="startOpenCv()" type="text/javascript"></script>
<script src="lodash.js"></script>
<body>
<video id="video-element"></video>
<canvas id="transfer-canvas" hidden></canvas>
<canvas id="output-canvas"></canvas>
<canvas id="contour-canvas" hidden></canvas>
<canvas id="lines-canvas" hidden></canvas>
<canvas id="intersection-points-canvas" hidden></canvas>
<canvas id="final-vertex-canvas"></canvas>
<iframe id="sim-frame"
src="quadrilateral_en_phet.html?deviceConnection&postMessageOnLoad"
width="640" height="480"></iframe>
<div>
<label id="min-h-label" for="min-h">Min h:</label>
<input type="range" id="min-h" min="0" max="255" value="0">
<label id="min-s-label" for="min-s">Min s:</label>
<input type="range" id="min-s" min="0" max="255" value="0">
<label id="min-v-label" for="min-v">Min v:</label>
<input type="range" id="min-v" min="0" max="255" value="0">
<label id="max-h-label" for="max-h">max h:</label>
<input type="range" id="max-h" min="0" max="255" value="150">
<label id="max-s-label" for="max-s">max s:</label>
<input type="range" id="max-s" min="0" max="255" value="150">
<label id="max-v-label" for="max-v">max v:</label>
<input type="range" id="max-v" min="0" max="255" value="150">
</div>
</body>
<script>
// connect to simulation
let simulationModel = null;
let dot = null;
let Vertex = null;
const iframe = document.getElementById( 'sim-frame' );
window.addEventListener( 'message', event => {
if ( !event.data ) {
return;
}
let data;
try {
data = JSON.parse( event.data );
}
catch( e ) {
return;
}
if ( data.type === 'load' ) {
simulationModel = iframe.contentWindow.simModel;
dot = iframe.contentWindow.phet.dot;
kite = iframe.contentWindow.phet.kite;
Vertex = iframe.contentWindow.phet.quadrilateral.Vertex;
}
} );
class Side {
constructor( p1, p2 ) {
this.p1 = p1;
this.p2 = p2;
this.length = Math.sqrt( ( p2.x - p1.x ) * ( p2.x - p1.x ) + ( p2.y - p1.y ) * ( p2.y - p1.y ) );
this.angle = Math.atan2( p2.y - p1.y, p2.x - p1.x ) + Math.PI / 2;
}
}
class LineWithAngle {
/**
* Construct a line where the angle of the line is already calculated from an opencv operation.
* @param p1
* @param p2
* @param theta
*/
constructor( p1, p2, theta ) {
this.line = new kite.Line( p1, p2 );
this.angle = theta;
}
}
const sliderIds = [
"min-h",
"min-s",
"min-v",
"max-h",
"max-s",
"max-v"
];
sliderIds.forEach( sliderId => {
const slider = document.getElementById( sliderId );
slider.addEventListener( 'input', () => {
document.getElementById( `${sliderId}-label` ).textContent = `${sliderId}: ${slider.value}`
} );
} )
// these values work OK to detect the green on my quad
// min: [63, 46, 87, 0]
// max: [91, 150, 246, 255]
const getMinFilterValues = () => {
return [
parseInt( document.getElementById( 'min-h' ).value, 10 ),
parseInt( document.getElementById( 'min-s' ).value, 10 ),
parseInt( document.getElementById( 'min-v' ).value, 10 ),
0 // alpha
]
}
const getMaxFilterValues = () => {
return [
parseInt( document.getElementById( 'max-h' ).value, 10 ),
parseInt( document.getElementById( 'max-s' ).value, 10 ),
parseInt( document.getElementById( 'max-v' ).value, 10 ),
255 // alpha
]
}
const startOpenCv = () => {
const width = 640;
const height = 480;
// Set up video capture with a webcam
let video = document.getElementById( "video-element" );
navigator.mediaDevices.getUserMedia( { video: true, audio: false } )
.then( function( stream ) {
video.srcObject = stream;
video.play();
} )
.catch( function( err ) {
console.log( "An error occurred! " + err );
} );
let transferCanvas = document.getElementById( "transfer-canvas" );
transferCanvas.width = width;
transferCanvas.height = height;
let context = transferCanvas.getContext( "2d" );
let imageCopyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let filterOutputMat = new cv.Mat( height, width, cv.CV_8UC4 );
let intersectionPointMat = new cv.Mat( height, width, cv.CV_8UC4 );
let finalVertexPointMat = new cv.Mat( height, width, cv.CV_8UC4 );
let copyTempMat = new cv.Mat( height, width, cv.CV_8UC4 );
let kMeansLabelsMat = new cv.Mat();
const clusterCount = 4;
let grayForContoursMat = new cv.Mat( height, width, cv.CV_8UC1 );
let contoursDestinationMat = new cv.Mat( height, width, cv.CV_8UC1 );
let greyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let cannyMat = new cv.Mat( height, width, cv.CV_8UC4 );
let erosionMat = new cv.Mat( height, width, cv.CV_8UC4 );
let dst = new cv.Mat( height, width, cv.CV_8UC1 );
const FPS = 30;
// colors for the color filter
// FOr a color that is not red this generally works. These higher and lower values match the green
// of some ducktape I bought.
// const lower = [ 44, 94, 90, 255 ];
// const higher = [ 88, 250, 239, 255 ];
// These values worked better at 4:34 pm light in my apartment
// const lower = [ 41, 120, 58, 255 ];
// const higher = [ 103, 255, 255, 255 ];
// These values worked better at 11:09 am in the light in my apartment
const lower = [ 48, 84, 58, 255 ];
const higher = [ 103, 255, 255, 255 ];
// set slider values to these defaults so we can tweak them after load
document.getElementById( 'min-h' ).value = lower[ 0 ];
document.getElementById( 'min-s' ).value = lower[ 1 ];
document.getElementById( 'min-v' ).value = lower[ 2 ];
document.getElementById( 'max-h' ).value = higher[ 0 ];
document.getElementById( 'max-s' ).value = higher[ 1 ];
document.getElementById( 'max-v' ).value = higher[ 2 ];
function processVideo() {
let begin = Date.now();
// now we rely on the dot library to find line intersections
if ( dot && kite ) {
// get the video data and draw to a temp canvas so that it can be read by cv functions
context.drawImage( video, 0, 0, transferCanvas.width, transferCanvas.height );
imageCopyMat.data.set( context.getImageData( 0, 0, width, height ).data );
// input canvas
let imageSource = cv.imread( 'transfer-canvas' );
let hsvSource = new cv.Mat();
cv.cvtColor( imageSource, hsvSource, cv.COLOR_RGB2HSV );
// use these instead to filter out values by slider
const modifiedLower = getMinFilterValues();
const modifiedHigher = getMaxFilterValues();
let lowMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), modifiedLower );
let highMat = new cv.Mat( hsvSource.rows, hsvSource.cols, hsvSource.type(), modifiedHigher );
// filter out to only detect light green of the hardware green
cv.inRange( hsvSource, lowMat, highMat, filterOutputMat );
// Try eroding first to get clearer lines
let M = cv.Mat.ones( 10, 10, cv.CV_8U );
let anchor = new cv.Point( -1, -1 );
cv.erode( filterOutputMat, erosionMat, M, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
cv.dilate( erosionMat, erosionMat, M, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
cv.imshow( 'output-canvas', erosionMat );
M.delete();
let contours = new cv.MatVector();
let hierarchy = new cv.Mat();
let tmpContoursMat = cv.imread( 'output-canvas' );
cv.cvtColor( tmpContoursMat, grayForContoursMat, cv.COLOR_RGB2GRAY );
cv.findContours( grayForContoursMat, contours, hierarchy, cv.RETR_CCOMP, cv.CHAIN_APPROX_SIMPLE );
tmpContoursMat.delete();
hierarchy.delete();
if ( contours.size() > 0 ) {
for ( let i = 0; i < contours.size(); i++ ) {
const rotatedRect = cv.minAreaRect( contours.get( i ) );
const area = rotatedRect.size.width * rotatedRect.size.height;
if ( area > 100 ) {
let vertices = cv.RotatedRect.points( rotatedRect );
for ( let i = 0; i < 4; i++ ) {
cv.line( erosionMat, vertices[ i ], vertices[ ( i + 1 ) % 4 ], [ 255, 0, 0, 255 ], 2, cv.LINE_AA, 0 );
}
}
}
}
cv.imshow( 'output-canvas', erosionMat );
// finally find the vertex positions
const markers = [];
if ( contours.size() > 0 ) {
for ( let i = 0; i < contours.size(); i++ ) {
if ( markers.length < 4 ) {
const rotatedRect = cv.minAreaRect( contours.get( i ) );
const area = rotatedRect.size.width * rotatedRect.size.height;
if ( area > 100 ) {
markers.push( rotatedRect );
}
}
}
}
markers.forEach( marker => {
cv.circle( finalVertexPointMat, marker.center, 5, [ 0, 255, 0, 255 ], cv.FILLED );
} );
cv.imshow( 'final-vertex-canvas', finalVertexPointMat );
// try to send the marker positions to the simulation
if ( markers.length === 4 ) {
// for now we don't label the vertices, find the min and max x, y values of each
// and use these to determine relative positions
// get centers and convert to Vector2 so we can use dot functions
const centers = markers.map( marker => new dot.Vector2( marker.center.x, marker.center.y ) );
// OpenCV goes has +y in the negative direction. We need to transform the y values so that the origin is
// at the bottom to pass the same values to the simulation model
const transformedMarkerPositions = centers.map( centerPosition => new dot.Vector2( centerPosition.x, height - centerPosition.y ) );
const xSorted = _.sortBy( transformedMarkerPositions, center => center.x );
const ySorted = _.sortBy( transformedMarkerPositions, center => center.y );
// the left vertices are the first elements of the xSorted array and the bottom vertices
// are the first elements of the ySorted array
let leftTopCenter;
let leftBottomCenter;
let rightTopCenter;
let rightBottomCenter;
if ( xSorted[ 0 ].y > xSorted[ 1 ].y ) {
leftTopCenter = xSorted[ 0 ];
leftBottomCenter = xSorted[ 1 ];
}
else {
leftTopCenter = xSorted[ 1 ];
leftBottomCenter = xSorted[ 0 ];
}
if ( ySorted[ 0 ].x > ySorted[ 1 ].x ) {
rightBottomCenter = ySorted[ 0 ];
}
else {
rightBottomCenter = ySorted[ 1 ];
}
if ( xSorted[ 2 ].y > xSorted[ 3 ].y ) {
rightTopCenter = xSorted[ 2 ]
}
else {
rightTopCenter = xSorted[ 3 ]
}
const deltaX = rightTopCenter.x - leftTopCenter.x;
const deltaY = rightTopCenter.y - leftTopCenter.y;
const angleRadians = Math.atan2( deltaY, deltaX );
if ( simulationModel ) {
// get lengths and angles from the positions here, using the existing model function to set the
// quadrilateral with external data
const topLength = rightTopCenter.distance( leftTopCenter );
const rightLength = rightTopCenter.distance( rightBottomCenter );
const bottomLength = rightBottomCenter.distance( leftBottomCenter );
const leftLength = leftBottomCenter.distance( leftTopCenter );
console.log( topLength, bottomLength );
// quadrilateral code has a function to get angles for us
const p1Angle = Vertex.calculateAngle( leftBottomCenter, leftTopCenter, rightTopCenter );
const p2Angle = Vertex.calculateAngle( leftTopCenter, rightTopCenter, rightBottomCenter );
const p3Angle = Vertex.calculateAngle( rightTopCenter, rightBottomCenter, leftBottomCenter );
const p4Angle = Vertex.calculateAngle( rightBottomCenter, leftBottomCenter, leftTopCenter );
simulationModel.markerRotationProperty.value = -angleRadians;
if ( simulationModel.isCalibratingProperty.value ) {
simulationModel.setPhysicalModelBounds( topLength, rightLength, bottomLength, leftLength );
}
else {
simulationModel.quadrilateralShapeModel.setPositionsFromLengthAndAngleData( topLength, rightLength, bottomLength, leftLength, p1Angle, p2Angle, p3Angle, p4Angle );
}
}
}
// memory cleanup
// clear lines output for next time
dst.delete();
dst = cv.Mat.zeros( height, width, cv.CV_8UC3 );
// clear circles output for next time
intersectionPointMat.delete();
intersectionPointMat = cv.Mat.zeros( height, width, cv.CV_8UC3 );
finalVertexPointMat.delete();
finalVertexPointMat = cv.Mat.zeros( height, width, cv.CV_8UC3 );
lowMat.delete();
highMat.delete();
hsvSource.delete();
contours.delete();
imageSource.delete();
}
// schedule next one.
let delay = 1000 / FPS - ( Date.now() - begin );
window.setTimeout( processVideo, delay );
}
// schedule first one.
window.setTimeout( processVideo, 0 );
};
</script>
</html>Demonstration of the behavior, with detected positions controlling the sim:
Don't have labelled vertices or something resilient to perspective figured out yet but I think that seems relatively straight forward to work on next. |
|
Discussed status with @BLFiedler, over slack he mentioned two things that would be good to have next
EDIT: For next time, cv has built in functions to flip an image |
|
I think we'll need to support both horizontal and vertical flip? I made a quick video about possible detection window orientations with respect to the marker locations WIN_20220309_11_27_07_Pro.mp4 |
|
These are really interesting perspectives on the device. I have questions about how to consistently start the description. |
|
When a shareable version is ready, let's keep a version with the non-identified vertices (relabels when the shape rotates) Next step will be adding vertex identification to enable rotation of the quad while keeping the same vertex from startup |
|
Played around with small squares affixed to the TMQ as well as free moving green blocks. I mounted my webcam on the ceiling above me (sloped ceiling). https://phet-dev.colorado.edu/html/jg-tests/opencv-test/ Setup notes:
1.) SUPER FUN. Amazing how much we can already do with it once it's set up. Quite smooth (with the exception of the below notes) when the parameters are dialed in. 2.) A few videos I took playing around with the current setup.
Video 1: OpenCV.Test.-.Google.Chrome.2022-04-20.09-37-31.mp4Video 2: OpenCV.Test.-.Google.Chrome.2022-04-20.09-47-51.mp4 |
|
@BLFiedler Sounds very cool! I can't seem to get the videos to load... is anyone else having this problem? |
|
I tried changing the formatting of the post above which made the embedding show up. Let me know if that fixes it. Otherwise, I've put the videos here, though they may take a bit to process: https://drive.google.com/drive/folders/1zwKRagycbptEeRXa3AhiuEQ0CeUCVsvh?usp=sharing |
|
The corner demo is so cool @BLFiedler! Do you know what is happening in the TMQ demo? It jitters without movement? |
|
Repost of my above comments with some additional details taken while chatting with @jessegreenberg . Includes plans for new issues to prioritize after Voicing:
|
|
Cool videos! I think this shows lots of potential, particularly with the four blocks... |
This was referenced May 3, 2022
|
Updating needs for OpenCV issues from
This was done as part of JGs tests - currently being further developed in: #141
This shouldn't be an issue when using 4 distinctly colored markers: #141
Also to be worked on in #141 as part of marker differentiation.
Creating new issue that retains last used values from browser cache.
On hold for now - current interface usable with PhET-iO and the sim can be full screen to hide the interface. Hiding this in a menu will make setup difficult. |
|
For current needs, this is complete. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Snippets taken/edited from Slack convo between BF & JG
BF: There's interest in at least exploring what it might take to implement the fiducial markers as a means of providing device orientation while CHROME lab does the same with the embedded sensors (gyroscopes/accelerometers).
BF: I think of greatest importance is considering the lift we'd be asking to pull in data from multiple sources and what would need to be communicated across the three: microcontroller/sim/marker detection.
Related repos and issues:
https://github.com/phetsims/tangible
Investigate other marker input strategies: phetsims/tangible#7
Performance implications of user testing with mechamarkers: phetsims/ratio-and-proportion#28
Will tangible input be included in the published simulation?: phetsims/ratio-and-proportion#89
JG: My biggest question right off the bat is if you have any thoughts about phetsims/tangible#7? It sounds like the marker strategy that was tried in RaP may not have been performant enough. Should we start with what was in RaP or just try something else?
BF: Yeah, so it was not quite enough to allow smooth movement when moving multiple objects. I'll see if I can get the demo up and running for Monday. It's possible with just one and no need for very rapid detection, it may not be so bad
BF: If we come up with enough questions, we can reach out to Peter from Ellen's lab to check on the current state of things and see what we can pull in re: rotation and tilt. They also had different sets of markers that seemed to possibly perform differently? And he had not updated in Spring 2021, but things might be different now?
JG: Just pondering internally... If we have a computer vision solution, will we still need the microcontroller?
BF: Yeah, I think this is only intended to partially solve the global orientation problem. I am absolutely positive using multiple markers will be nothing but trouble regarding constant detection/occlusion and resulting hitches in model updates for the existing Quad device from CHROME (based on the form factor and how a user moves their hands around the device). But, if we are only relying on ~1 to tell us if something is positioned upright or not, it may not matter too much. Of note, if we introduce any markers at all we will have to accept there will be some moments of desync between the visual display and what the learner is doing whenever detection is lost (be it occlusion, tilt, or goblins).
We should consider the implications of both scenarios:
Tagging @zepumph, since he has worked most extensively, in case there are any nuggets of wisdom to share after talking to @jessegreenberg .
@BLFiedler & @jessegreenberg meeting on 11/1 to discuss.
The text was updated successfully, but these errors were encountered: