docs/photogrammetry_bundle_adjustment_structure_from_motion_reprojection_error.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d1288540-9d8e-47f4-a907-6321abd4fca1",
   "metadata": {},
   "source": [
    "photogrammetry is the broad field encompassing techniques for extracting measurements from photographs. Structure from Motion is a specific approach within photogrammetry that uses image sequences to build 3D models. Bundle Adjustment is a mathematical technique used within both photogrammetry and SfM to refine these models and improve their accuracy.\n",
    "\n",
    "\n",
    "## Photogrammetry\n",
    "\n",
    "   - **Definition**: Photogrammetry is a technique used to measure and interpret features from photographs. It's broadly used in various fields such as surveying, mapping, engineering, and archaeology.\n",
    "   - **Application**: It involves taking measurements from photographs to create maps, 3D models, or drawings of real-world objects or scenes.\n",
    "   - **Process**: This process can use either single or multiple photographs and relies on the principles of perspective and projective geometry.\n",
    "   - **Tools & Techniques**: It might involve various software and hardware tools, like drones for aerial photography, and specialized software for image processing and 3D reconstruction.\n",
    "\n",
    "## Structure from Motion (SfM)\n",
    "\n",
    "   - **Definition**: SfM is a photogrammetric method used to create 3D structures from 2D image sequences. It's a subset of computer vision and photogrammetry.\n",
    "   - **Application**: Primarily used to reconstruct 3D models from photographs, particularly in applications where traditional surveying methods are impractical.\n",
    "   - **Process**: The process involves identifying and matching features across a series of images, estimating camera positions and angles, and then reconstructing a 3D structure based on these estimations.\n",
    "   - **Relation to Photogrammetry**: SfM is more automated and software-driven compared to traditional photogrammetry and is considered a modern advancement in the field.\n",
    "\n",
    "## Bundle Adjustment\n",
    "   - **Definition**: Bundle Adjustment is a mathematical optimization technique used in computer vision and photogrammetry.\n",
    "   - **Application**: It's used to refine a visual reconstruction to produce jointly optimal 3D structure and viewing parameter (camera position, orientation) estimates.\n",
    "   - **Process**: It adjusts the 3D coordinates of the point cloud and camera parameters by minimizing the re-projection error, which is the difference between the observed image points and projected points from the 3D model.\n",
    "   - **Relation to SfM and Photogrammetry**: Bundle Adjustment is often the final step in the SfM process to refine the model. It's a critical part of both SfM and photogrammetry when accuracy is essential."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbe13ae2-fe90-4298-bf90-6b2a7328c081",
   "metadata": {},
   "source": [
    "Ref: [1](https://www.cs.cornell.edu/~snavely/bundler/bundler-v0.4-manual.html), [2](https://www.eecs.umich.edu/courses/eecs442-ahowens/fa21/slides/lec22-sfm.pdf), [3](https://ceres-solver.googlesource.com/ceres-solver/+/1.12.0/examples/snavely_reprojection_error.h)\n",
    ",[4](http://ceres-solver.org/nnls_tutorial.html#bundle-adjustment), [5](https://homes.cs.washington.edu/~sagarwal/bal.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d5e3552-c3b3-47dc-93e2-304183fba1b0",
   "metadata": {},
   "source": [
    "## CMVS-PMVS\n",
    "Refs: [1](https://github.com/pmoulon/CMVS-PMVS)\n",
    "\n",
    "\n",
    "## openMVG\n",
    "Refs: [1](https://opensourcephotogrammetry.blogspot.com/)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "##  Pipeline for SFM\n",
    "\n",
    "**Feature Detection and Matching:**\n",
    "First, features are detected in each image. Common algorithms include SIFT, SURF, or ORB. \n",
    "These features are then matched between images to find correspondences. OpenCV's cv::BFMatcher or cv::FlannBasedMatcher can be used for this.\n",
    "\n",
    "**Pose Estimation:**\n",
    "With matched features, you estimate the relative pose between two images. cv::recoverPose is often used here. It computes the rotation and translation between two views given corresponding points and the essential matrix. The essential matrix can be calculated using `cv::findEssentialMat`.\n",
    "\n",
    "```cpp\n",
    "cv::Mat E = cv::findEssentialMat(points1, points2, focal, pp, cv::RANSAC, 0.999, 1.0, mask);\n",
    "cv::recoverPose(E, points1, points2, R, t, focal, pp, mask);\n",
    "```\n",
    "\n",
    "\n",
    "**Incremental or Global Structure from Motion:**\n",
    "- Incremental SfM: Start with an initial pair of images, recover pose, and then add images one by one, adjusting the structure and camera poses with each addition. Here, after using cv::recoverPose, you would typically use a bundle adjustment step to refine the structure and camera poses.\n",
    "- Global SfM: All images are considered at once. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}