中文版

Low-poly Reconstruction, which we might also call Vectorized 3D Reconstruction or Structured Reconstruction, centers on extracting structural, topological, and semantic information from scattered, unordered, and redundant dense point clouds or triangular meshes. Its core objective is to represent a model using points, lines, and planes (and their combinations) defined by analytical mathematical equations.

When the final model is completely composed of compact Planar Polygons, we have achieved Low-poly Reconstruction. Unlike traditional surface fitting, low-poly reconstruction pursues extreme vertex decimation and the preservation of structural features, all while maintaining geometric fidelity.

But to truly understand low-poly reconstruction, we can’t just talk about low-poly reconstruction (jokes aside). Looking back at the history of 3D reconstruction, the technological evolution can be cleanly divided into the following five stages:

1. Point Acquisition: Sensing the Physical World

The foundation of all reconstruction starts with acquiring spatial coordinates $(x, y, z)$. The primary goal of this stage is to use sensors or multi-view geometry techniques to discretize real-world physical surfaces into point sets in digital space.

  • Hardware-driven: ToF (Time of Flight) cameras, structured light scanners.
  • Algorithm-driven: Epipolar geometry, SfM (Structure from Motion). Classic open-source solutions like COLMAP have massively propelled image-based sparse point cloud generation.

2. Point Reconstruction: From Local to Global

A single sensor or a single scan can only capture local point cloud fragments. The task here is to register and fuse point clouds from different viewpoints, eliminating redundancy and reducing scanning errors to form a globally consistent representation of the scene.

  • Classic Algorithms: KinectFusion, a milestone for real-time 3D reconstruction based on depth cameras, alongside various ICP (Iterative Closest Point) variants.
  • Limitations: The fused point cloud remains an unordered, everywhere non-differentiable discrete set. Even with naive spatial partitioning using KNN or octrees, true topological connections cannot be established, making it difficult to support downstream geometric operations.

3. Dense Surface Reconstruction: Building Continuous Manifolds

To perform physical simulations or extract elevation data in digital space, discrete point clouds must be converted into continuous, differentiable meshes with topological correlations. Taking the classic Stanford Bunny as an example, converting it from a point cloud into a surfaced model is the core task of this stage.

  • Classic Algorithms: Poisson Surface Reconstruction frames surface reconstruction as a global optimization problem solving the Poisson equation; alternatively, computational geometry-based Delaunay Triangulation.
  • The Pain Point: Massive data volume. For dense reconstructions of city-scale scenes, vertex counts often hit the billions. Assuming a single vertex contains coordinates (3 doubles, 24 Bytes), normals (3 floats, 12 Bytes), and colors (3 chars, 3 Bytes), 3 billion vertices will consume roughly 110 GB of RAM, ignoring byte alignment. This is an absolute disaster for storage, transmission, and online rendering.

4. Geometric Simplification & Approximation: Alleviating the Data Explosion

To tackle the compute and storage challenges brought by dense meshes, mesh simplification algorithms emerged. Their goal is to drastically reduce the number of faces while preserving as much geometric detail and visual characteristics as possible.

  • Classic Algorithms: QEM (Quadric Error Metrics) is the quintessential edge-collapse simplification algorithm (its core idea is to determine the collapse order by minimizing the sum of squared distances from a vertex to its adjacent planes); VSA (Variational Shape Approximation) approximates the original geometry by extracting anisotropic Planar Proxies.
  • The Bottleneck: For large-scale buildings or macro urban scenes, these algorithms still generate “unstructured” triangular faces, and the polygon count remains massive. This fails to meet the machine’s need for a “high-level understanding” of the scene.

5. Low-poly & Vectorized Reconstruction: Further Slashing Data Volume

Where there are pain points, there is demand; where there is demand, solutions emerge. To satisfy high-level tasks that require computers to “understand” geometry—such as real-time processing systems, online rendering, robotic manipulation, autonomous driving, and surveying—algorithms need to further slash data volume while guaranteeing precision. Thus, low-poly reconstruction took the stage.

Low-poly reconstruction outputs a finite, compressed surface composed purely of polygons. Its core paradigm typically involves:

  1. Primitive Extraction: Extracting geometric primitives like planes and cylinders.
  2. Primitive Assembly: Determining the intersection relationships, boundary lines, and vertices between primitives.
  3. Optimization: Finding the optimal solution that balances geometric fidelity and model compactness.

Key Algorithms in Low-poly Reconstruction

[16’ ECCV] Manhattan-world Urban Reconstruction from Point Clouds Project Page: https://github.com/LiangliangNan/ManhattanModeler
It generates candidate cuboids from point clouds via non-uniform space partitioning and uses Markov Random Field (MRF) global optimization to filter out the optimal set of aligned cuboids. It robustly approximates and reconstructs urban building geometry under the Manhattan-world assumption without needing prior instance segmentation.
MANHATON1

[17’ ICCV] PolyFit: Polygonal Surface Reconstruction from Point Clouds Project Page: https://github.com/LiangliangNan/PolyFit
It extracts planar primitives and intersects them to partition space (generating candidate polygonal faces). It then uses Integer Linear Programming (ILP) for global optimization to strike a balance between data fitting, point coverage, and model compactness, robustly reconstructing lightweight polygonal surfaces from point clouds.
polyfit

[20’ ToG] Kinetic Shape Reconstruction (KSR) Project Page: https://www-sop.inria.fr/members/Florent.Lafarge/
Proposes a surface reconstruction method based on a Kinetic Data Structure. It allows planar polygons extracted from point clouds to expand at a constant speed until they collide, constructing a lightweight polyhedral space partition. Finally, a Min-cut algorithm extracts a compact, watertight, CAD-style polygonal mesh. It’s a very elegant algorithm with fewer space partitions compared to PolyFit.
ksr
As seen, the algorithm restores main structures well. However, fine details like the roof gutters and indoor ceiling beams cannot be recovered due to missing point cloud data.
ksr2
For the Meeting Room scene in the Tanks and Temples dataset, the data scale is abstracted from 3.1M points to a watertight model represented by roughly 3K vertices and 1.5K planar polygons.
ksr3
For the Barn scene, it abstracts 613K points into a watertight model with around 500 vertices and 200 planar polygons.

[21’ ISPRS] Vectorized Indoor Surface Reconstruction from 3D Point Cloud with Multistep 2D Optimization Project Page: https://github.com/3dv-casia/VecIM
Proposes a multi-step vectorized indoor reconstruction method (VecIM). It breaks the Manhattan or Atlanta world assumptions, reducing complex 3D structural extraction into a series of 2D arrangement-based global optimization problems (combining ILP and MRF) to robustly reconstruct models from heavily occluded indoor point clouds.

[22’ SIGGRAPH] Low-poly Mesh Generation for Building Models Project Page: https://lowpoly-modeling.github.io/
Introduces a visual metric-guided, three-stage automated algorithm (generating visual hulls, boolean carving of redundant structures, and progressive simplification) to automatically convert complex high-poly 3D building models into visually high-fidelity, watertight, and self-intersection-free low-poly meshes.
poly

[23’ ToG] Robust Low-Poly Meshing for General 3D Models Project Page: https://robust-low-poly-meshing.github.io/
Presents a robust re-meshing method that automatically converts any complex high-precision 3D model into a visually accurate, watertight, manifold, and self-intersection-free low-poly mesh, integrating seamlessly into existing 3D asset pipelines.

[24’ ECCV] Concise Plane Arrangements for Low-Poly Surface and Volume Modelling Project Page: https://github.com/raphaelsulzer/compod
By introducing plane insertion sorting and directly utilizing input points to guide local space partitioning, this paper successfully breaks the computational bottleneck of traditional plane arrangement algorithms. It achieves highly efficient reconstruction of minimalist, watertight 3D low-poly surfaces and lightweight volume models from massive point clouds.
CONSICE

[24’ ECCV] WindPoly: Polygonal Mesh Reconstruction via Winding Numbers Proposes a robust reconstruction scheme that discards traditional normal-vector dependency. It iteratively extracts planes from discrete point clouds to build a convex polyhedral space, then innovatively introduces Polygonal Winding Numbers for globally consistent orientation. This allows it to extract high-quality, highly detailed low-poly structural meshes even in the face of severe noise and missing data.
WINDPOLY

[26’ 3DV] Hierarchical Space Partition for Surface Reconstruction Project Page: https://hsr-3dv.github.io/
First restores missing planes in the scene and utilizes visibility for plane classification. Building on dynamic data structures (KSR), it proposes a novel Hierarchical Space Partition strategy: it replaces traditional global exhaustive searches or simple dynamic collision intersection calculations with an adaptive multi-scale hierarchical structure. This significantly reduces computational complexity and memory consumption, efficiently and robustly reconstructing polygonal surfaces that preserve sharp features and topological consistency.
As seen in the paper’s results, it recovers local details like the window sills in the Barn scene and details in the Meeting Room. It also demonstrates great generalization, showing impressive reconstruction results for free-form objects.
hsr

[26’ 3DV] NURBSFit: Robust Fitting of NURBS Surfaces to Point Clouds Introduces a robust fitting method called NURBSFit. By optimizing control points and knot vectors, it extracts and fits Non-Uniform Rational B-Spline (NURBS) parametric surfaces directly from noisy and incomplete 3D point clouds, generating highly smooth, precise, and continuous CAD surface models that meet industrial standards.
nurb1

Final Thoughts

Give Data to the Model, Not the Model to the Data

As we’ve seen, traditional reconstruction algorithms often carry a heavy flavor of “parameterization” or even “alchemy”: we need to manually design hand-crafted features and frantically fine-tune various thresholds based on experience. It feels a bit like forcing the issue, violently squeezing the desired geometric model out of the data. But hey, as long as there is compression, there is intelligence; manual compression is still intelligence (sarcasm noted).

In this era where compute and data are soaring, changing the mindset to—“Give data to the model, rather than the model to the data” (Sci-Fi author Liu Cixin would have no objections)—allowing algorithms to figure out the intrinsic patterns of the data through learning, often works miracles. In fact, academia has already seen a surge in deep learning-based low-poly reconstruction methods. We’ll talk about those when I have time.

How Does the Computer Understand?

Up to this point, we’ve successfully abstracted massive city-scale scenes into vectorized polygon models. The significance of this step isn’t just geometric simplification; it’s a “Memory Liberation Movement”. The computer can load it entirely into RAM at once, instead of flirting with the edge of crashing when facing hundreds of GBs of raw dense point clouds, barely surviving on frequent memory swapping (Virtual Memory: “No worries mate, theoretically I’m as big as your hard drive”).

However, merely stopping at the rendering level is far from enough. To a computer, standard mesh models (like OBJ, STL, PLY) are essentially just “Polygon Soup”. They only record low-level geometric and topological information (vertices, edges, faces), lacking physical properties, topological hierarchies, and functional semantics. If we want the computer to genuinely “understand”, parametrically manipulate, and deeply edit them, we must cross the chasm from “Computer Vision 3D Reconstruction” to “Computer-Aided Design (CAD) & Building Information Modeling (BIM)”.

Mainstream software like AutoCAD and SolidWorks don’t rely on polygon meshes at their core; instead, they are based on precise Boundary Representation (B-Rep). This representation uses analytical surfaces (exact planes, cylinders) and NURBS, combined with Constructive Solid Geometry (CSG) feature trees. The current academic approach is to use Mesh Segmentation and Primitive Fitting algorithms to convert simplified planar meshes into B-Rep solids with analytical expressions, thereby recovering the model’s mechanical parametric editing capabilities.

In the reconstruction of urban or indoor architectural scenes, the prevailing research direction is “Scan-to-BIM”. BIM is essentially a relational spatial database rich in semantics. A massive amount of literature and ongoing work is dedicated to using algorithms (like deep learning-based 3D semantic segmentation) to classify simplified geometric polygons. It doesn’t just extract the geometric boundaries of a plane, but accurately identifies it as a specific architectural component (e.g., wall, floor, ceiling, doors, and windows).

Once a simplified planar model is endowed with semantics, how do we feed it to the computer for full parsing? This requires domain-specific structured modeling languages and standard data schemas.

The industry doesn’t use loose general data structures; they have established highly structured underlying file standards: for instance, the STEP (ISO 10303) standard in mechanical CAD, IFC (Industry Foundation Classes) in BIM, and CityGML for urban digital twins.

These structured files are typically based on XML or EXPRESS data modeling languages, strictly extending the logical scope of fundamental polygon data. For example, during conversion, a specific simplified planar polygon is no longer just a geometric face; it’s encapsulated as an IfcWallStandardCase object. The file defines not only its 3D coordinates but also attaches semantic information (load-bearing properties, materials) and topological relationships (e.g., this wall “contains” a window and is “connected” to the floor).

The computer only needs to invoke standard library parsing interfaces at the lower level (like C++) to deserialize the geometric data into an entity object tree with logical correlations. At this point, the colossal 3D point cloud is truly transformed into a structured digital twin asset that the computer can fully comprehend and edit.