EndoS³: Endoscopic Semantic Splatting SLAM

Abstract

Purpose. Despite advances in 3D surgical scene understanding, semantic simultaneous localisation and mapping (SLAM) remains under-explored for two key reasons. First, it requires solving three challenging tasks in parallel: camera pose tracking, high-fidelity deformable reconstruction, and semantic mapping, making progress particularly demanding. Second, progress is further limited by the scarcity of datasets that provide the ground truth needed to evaluate all three tasks comprehensively under procedural conditions that closely reflect real endoscopic environments.

We propose EndoS³, a semantic SLAM framework based on 3D Gaussian splatting that uses per-frame instance masks to guide all three tasks. An instance-level motion decomposition separates static from dynamic instances, testing the flow that camera motion cannot explain against both flow noise and neighbouring instances. The camera is tracked against the static Gaussians, while an instance-aware warp field gives each dynamic instance its own control nodes and deformation network. A semantic node allocation redistributes a fixed node budget across instances according to their size and non-rigidity. We also introduce CRCD++, an ex-vivo cholecystectomy dataset with RGB-D frames, camera poses and instance annotations for full semantic SLAM evaluation.

Results. We compare EndoS³ with state-of-the-art general-purpose and surgical semantic SLAM methods in tracking, reconstruction and semantic mapping on the CRCD++ and the SemSup datasets. EndoS³ surpasses all previous approaches.

Conclusion. Modelling motion and deformation with semantic information supports accurate semantic SLAM in deformable surgical scenes. EndoS³ and CRCD++ provide a method and a benchmark for improved geometric and semantic understanding in endoscopic environments.

Method

Instance-level motion decomposition

Optical flow from WAFT, minus the flow explained by a PnP camera estimate, is pooled in superpixels split along instance boundaries. Each instance is tested against flow noise and against its neighbours; only instances that pass both are dynamic. The camera is tracked against the static part.

Instance-aware warp field

Every instance gets its own control nodes and deformation network, and motion only propagates within an instance, so instruments and tissue no longer blend at their contact points.

Semantic node allocation

A fixed node budget is redistributed across instances every frame, weighted by visible area and non-rigidity, so nodes follow instruments entering the view and regions that deform most.

CRCD++ dataset

20 robotic cholecystectomy sequences with camera poses, depth, and dense liver, gallbladder and tool masks on every frame.

Annotation comparison on CRCD frames: input image, eCRCD Track Anything propagated masks, and the CRCD++ manual annotations. — Input | eCRCD (Track Anything propagation) | CRCD++ manual annotation.

Sem.: semantic classes (^ssparse, ^ttools only, ^ntno tools).
	CholecSeg8k	EndoVis18	Endoscapes	Semantic-SuPer	StereoMIS	D4D	eCRCD	CRCD++
Poses	No	No	No	No	Yes	Yes	Yes	Yes
Depth	No	No	No	No	No	GT	No	Pseudo-GT
Sem.	13	12^s	6^s	3	1^t	1^t	2^nt	4
Seq. len.	80	Sparse	Sparse	150	1.4–9.6k	~3k	120	0.2–2.1k
Frames	8,080	2,384	493	600	~58.7k	300k+	–	14,688

Results

EndoS³ against six semantic SLAM methods on CRCD++ and Semantic-SuPer.

Visual comparison

Drag the divider to compare; click a thumbnail to swap the left side.

Left Right

Quantitative results

Camera trajectories

Drag to rotate, scroll to zoom.

Compare with

Loading 3-D viewer…

Ablations

Module ablation, CRCD++ C3. MD: motion decomposition · IW: instance-aware warp field · NA: semantic node allocation.
MD	IW	NA	PSNR ↑	Depth ↓	ATE ↓	mIoU ↑
✗	✓	✓	22.18	5.85	2.91	52.8
✓	✗	✓	23.28	4.62	2.55	71.3
✓	✓	✗	23.08	4.43	2.83	70.9
✓	✓	✓	24.35	4.35	2.29	74.4

Motion-decomposition tests, CRCD++ C3. z^A: instance test · z^B: boundary test.
z^A	z^B	PSNR ↑	ATE ↓	mIoU ↑
✓	✗	24.39	3.03	71.9
✗	✓	24.29	2.80	72.6
✓	✓	24.45	2.75	73.4

BibTeX

@misc{sheikhzeinoddin2026endos3,
  title  = {EndoS$^3$: Endoscopic Semantic Splatting SLAM},
  author = {Sheikh Zeinoddin, Mona and Wright, Benjamin and Tandogdu, Zafer
            and Shaw, Greg and Mazomenos, Evangelos and Stoyanov, Danail},
  year   = {2026}
}