What’s new

6.04.16We published evaluation results for Sparse Sampling Matting [14].
15.12.151) New sequences with natural hairs including 3 public sequences.
2) New metrics of temporal coherency chosen by careful analysis (see [3]).
3) New trimap generation method (more natural-looking and accurate).
4) Better ground-truth quality owing to correction of lightning changes during capturing.
5) Improvement in website loading speed and interface.
7.09.15We published the paper with our benchmark description [3].
30.12.14We published results for multiple levels of trimaps; use drop-down menu at the top left corner to switch levels.
29.12.14We added general ranking to the rating table.
10.11.14“Sparse codes as Alpha Matte” was added.
26.09.14Source sequences are available for online view now. Full screen mode was added.
30.08.14Composite sequences are available now.
27.08.14“Refine edge tool in Adobe After Effects” was added.
25.08.14The official opening.

Overview

Introduction

The Video­Mat­ting pro­ject is the first pub­lic ob­jec­tive bench­mark for video-mat­ting meth­ods. It con­tains scat­ter plots and rat­ing ta­bles for dif­fer­ent qual­ity met­rics. In ad­di­tion, re­sults for par­tic­i­pat­ing meth­ods are avail­able for view­ing on a player equipped with a mov­able zoom re­gion. We be­lieve our work will help rank ex­ist­ing meth­ods and aid de­vel­op­ers of new meth­ods in im­prov­ing their re­sults.

Datasets

The data set con­sists of five mov­ing ob­jects cap­tured in front of a green plate and seven cap­tured us­ing the stop-mo­tion pro­ce­dure de­scribed be­low. We com­posed the ob­jects over a set of back­ground videos with var­i­ous lev­els of 3D cam­era mo­tion, color bal­ance, and noise. We pub­lished ground-truth data for two stop-mo­tion se­quences and hid the rest to en­sure fair­ness of the com­par­i­son.

Us­ing thresh­old­ing and mor­pho­log­i­cal op­er­a­tions on ground-truth al­pha mattes, we gen­er­ated nar­row trimaps. Then, we di­lated the re­sults us­ing graph­cut-based en­ergy min­i­miza­tion which pro­vides us with more hand­made-look­ing trimaps than com­mon mor­pho­log­i­cal di­la­tion.

Chroma Keying

green screen
stop motion
Alpha mattes from chroma keying and stop-motion capture for the same image region. The stop-motion result is significantly better at preserving details.

Chroma key­ing is a com­mon prac­tice of the cin­ema in­dus­try: the cin­e­matog­ra­pher cap­tures an ac­tor in front of a green or blue screen, then the VFX ex­pert re­places the back­ground us­ing spe­cial soft­ware. Our eval­u­a­tion uses five green-screen video se­quences with a sig­nif­i­cant amount of semi­trans­parency (e.g., hair or mo­tion blur), pro­vided to us by Hol­ly­wood cam­era work. We ex­tract al­pha mattes and cor­re­spond­ing fore­grounds us­ing The Foundry Key­light. Chroma key­ing en­ables us to get al­pha mattes of nat­ural-look­ing ob­jects with ar­bi­trary mo­tion. Nev­er­the­less, this tech­nique can’t guar­an­tee that the al­pha maps are nat­ural, be­cause it as­sumes the screen color is ab­sent from the fore­ground ob­ject. To get al­pha maps that have a very nat­ural ap­pear­ance, we use the stop-mo­tion method.

Stop Motion

One-step capture over different backgrounds. We use checkerboard backgrounds instead of solid ones to eliminate screen reflection.

We de­signed the fol­low­ing pro­ce­dure to per­form stop-mo­tion cap­ture: A fuzzy toy is placed on the plat­form in front of an LCD mon­i­tor. The toy ro­tates in small, dis­crete steps along a pre­de­fined 3D tra­jec­tory, con­trolled by two ser­vos con­nected to a com­puter. Af­ter each step the dig­i­tal cam­era in front of the setup cap­tures the mo­tion­less toy against a set of back­ground im­ages. At the end of this process, the toy is re­moved and the cam­era again cap­tures all of the back­ground im­ages.

We paid spe­cial at­ten­tion to avoid­ing re­flec­tions of the back­ground screen in the fore­ground ob­ject. These re­flec­tions can lead to false trans­parency that is es­pe­cially no­tice­able in non­trans­par­ent re­gions. To re­duce the amount of re­flec­tion we used checker­board back­ground im­ages in­stead of solid col­ors, thereby ad­just­ing the mean color of the screen to be the same for each back­ground.

At the end we cor­rected global light­ing changes caused by light bulb flick­er­ing. Thus fi­nally we ob­tain al­pha mattes with less than 1% of noise level. The de­tailed de­scrip­tion of ground-truth ex­trac­tion meth­ods is given in [3].

Evaluation Methodology

Our com­par­i­son in­cludes both im­age- and video-mat­ting meth­ods. We ap­ply each mat­ting method to the videos in our data set, and then com­pare the re­sults us­ing the fol­low­ing met­rics of per-pixel ac­cu­racy and tem­po­ral co­herency (look into our pa­per [3] for com­par­i­son of dif­fer­ent met­rics):

Equationmultiline equation
Equationmultiline equation
Equationmultiline equation

Here Equationnumber-signde­notes to­tal num­ber of pix­els, Equationalpha Subscript p comma tand EquationModifyingAbove alpha With caret Subscript p comma tde­note trans­parency val­ues of video mat­ting un­der con­sid­er­a­tion and ground truth cor­re­spond­ingly at pixel Equationpof frame Equationt, and Equationv Subscript pde­notes mo­tion vec­tor at pixel Equationp. We use op­ti­cal-flow al­go­rithm [11] com­puted for ground-truth se­quences. It is worth not­ing that mo­tion-aware met­rics will not give un­fair ad­van­tage to mat­ting meth­ods based on the sim­i­lar mo­tion es­ti­ma­tion method since they do not have ground truth se­quence. The de­tailed de­scrip­tion of used qual­ity met­rics is given in [3].

Public Sequences

For the train­ing pur­poses we pub­lish here three test se­quences with their ground-truth trans­parency maps. De­vel­op­ers and re­searchers are wel­come to use these se­quences, but we ask to cite us [3]

Participate

We in­vite de­vel­op­ers of video-mat­ting meth­ods to use our bench­mark. We will eval­u­ate the sub­mit­ted data and re­port scores to the de­vel­oper. In cases where the de­vel­oper specif­i­cally grants per­mis­sion, we will pub­lish the re­sults on our site. We can also pub­lish anony­mous scores for blind-re­viewed pa­pers. To par­tic­i­pate, sim­ply fol­low these steps:

  1. Download the data set containing our sequences: City, Flowers, Concert, Rain, Snow, Vitaliy, Artem, Slava, Juneau, Woods,
  2. Apply your method to each of our test cases
  3. Upload the alpha and foreground sequences to any file-sharing service. We kindly ask you to maintain these naming and directory-structure conventions. If your method doesn't explicitly produce the foreground images you can skip uploading them; in this case, we will generate them using method proposed in [7].
  4. Fill in this form to provide information about your method

Con­tact us by email with any ques­tions or sug­ges­tions at ques­tions@video­mat­ting.com.

Cite Us

To re­fer to our eval­u­a­tion or test se­quences in your work cite our pa­per [3].

                
@inproceedings{Erofeev2015,
	title={Perceptually Motivated Benchmark for Video Matting},
	author={Mikhail Erofeev and Yury Gitman and  Dmitriy Vatolin and Alexey Fedorov and Jue Wang},
	year={2015},
	month={September},
	pages={99.1-99.12},
	articleno={99},
	numpages={12},
	booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
	publisher={BMVA Press},
	doi={10.5244/C.29.99},
	isbn={1-901725-53-7},
	url={https://dx.doi.org/10.5244/C.29.99}
}
                

Evaluation

Rating

Trimap size:
Trimap available for each frame
year
rank
city
rain
concert
flowers
snow
Slava
Vitaliy
Artem
juneau
woods
Bayesian Matting [2]20017.958.76951.66810.795144.53830.36853.28843.70870.50996.697140.129
Robust Matting [12]20074.537.54626.1519.61288.15418.52533.31634.15346.91778.56680.415
Refine Edge [15]20135.837.78738.4868.641129.56627.47731.36536.65738.754102.02894.507
Closed Form [7]20083.931.34231.80310.967149.48915.86223.67330.15130.60269.94482.136
Learning Based [13]20092.529.98133.08410.846134.82715.23123.35131.46230.20158.38156.601
Nonlocal mattings [5]20119.754.21892.371033.8610243.471145.131064.051061.751067.308135.0810171.7910
Shared Matting [4]20104.936.79546.72710.30487.64324.41635.18735.26545.23668.52367.173
Comprehensive Samplings [10]20084.034.48430.18212.10879.60116.17325.02435.25441.05577.72570.184
KNN Matting [1]20128.967.831092.22964.7311112.46532.96961.50950.87979.2010113.419123.638
Spectral Mattings [6]201210.786.3211105.241113.159210.041048.761165.721168.8311109.4411157.1811182.8211
Sparse Samplings [14]20163.233.42333.5059.94381.06218.42423.64235.83638.00365.17264.122
Bayesian Matting [2]20018.973.18967.92816.069157.28939.59960.37856.05985.5310115.889169.339
Robust Matting [12]20075.648.19732.89314.72799.81425.96542.51638.88559.08793.79696.136
Refine Edge [15]20135.542.23641.74610.451127.02631.40735.08539.46642.914108.71797.767
Closed Form [7]20083.734.53335.86413.354154.12821.64427.19333.01135.45281.48392.755
Learning Based [13]20092.733.44237.63513.233138.31719.95326.51234.51234.70168.41163.531
Nonlocal mattings [5]20119.655.04897.85931.8110242.751146.941067.111065.091069.018144.6610176.6110
Shared Matting [4]20105.441.44556.75714.14596.71327.66642.92740.05758.17683.02478.914
Comprehensive Samplings [10]20083.338.43428.29114.45683.85116.84127.29437.07347.38589.96578.453
KNN Matting [1]20128.679.351098.711066.2411112.81534.60864.17954.44884.389112.138128.868
Spectral Mattings [6]201210.699.0511125.291116.048214.421059.611174.551177.2511122.8911162.7311188.1611
Sparse Samplings [14]20162.133.20131.92212.34285.52217.66224.64137.78439.87371.05269.942
Bayesian Matting [2]20019.398.131095.95838.5410170.76959.511080.721067.649122.089135.599179.259
Robust Matting [12]20076.467.71849.02519.875109.79444.10862.18750.54681.497118.097115.967
Refine Edge [15]20135.250.31451.81615.852125.25640.73745.38546.43552.224118.888103.305
Closed Form [7]20084.145.04341.69324.197159.47830.09436.85439.20146.972104.073110.266
Learning Based [13]20093.044.97243.75421.906143.54726.50333.89240.44245.93190.35281.221
Nonlocal mattings [5]20119.056.766113.901033.979241.311150.74977.14968.371076.436159.0810181.1410
Shared Matting [4]20105.555.84579.09719.784107.45332.94560.78651.477122.3310107.34497.224
Comprehensive Samplings [10]20083.456.79737.36216.96387.64124.04234.32343.69364.635110.50593.353
KNN Matting [1]20127.895.899112.70966.1211113.10537.18671.26860.03898.428110.876138.668
Spectral Mattings [6]201210.6116.8711157.371126.128218.941076.221188.741191.6911148.7011169.3311193.4611
Sparse Samplings [14]20161.743.32134.66114.78189.79219.23129.92144.59451.94385.44181.772
Bayesian Matting [2]20019.135.93965.2489.287104.681027.61835.041146.41860.341088.4610121.9110
Robust Matting [12]20074.322.32532.7317.26347.27816.04418.72634.47324.96453.42642.403
Refine Edge [15]20133.914.41144.0856.30240.69518.36616.53234.93417.83253.43743.605
Closed Form [7]20082.816.37241.2937.67435.14414.59216.40132.43117.37151.40347.717
Learning Based [13]20092.817.29342.9947.88634.03214.44117.07434.37218.37346.02139.462
Nonlocal mattings [5]20119.629.078110.76914.1010120.011129.821032.69959.911036.229108.431196.829
Shared Matting [4]20105.622.72659.5977.72545.92720.34722.35738.02729.56747.45238.191
Comprehensive Samplings [10]20084.822.78739.97213.83933.11115.61317.51537.09525.72553.33545.046
KNN Matting [1]20128.836.1610115.081025.581144.09627.80931.61856.08934.80887.53975.368
Spectral Mattings [6]20129.465.5011131.74114.781103.61939.031134.391063.211164.401187.448122.5311
Sparse Samplings [14]20164.921.78444.8569.86834.07316.94516.55337.40625.84651.77442.984
Bayesian Matting [2]20019.646.961077.27811.327111.601033.851039.541156.86875.5011102.7010151.6011
Robust Matting [12]20075.426.36637.7328.35552.07819.18521.45637.08428.55657.78747.955
Refine Edge [15]20133.415.24147.3566.58240.54519.40616.77235.74218.24254.11543.433
Closed Form [7]20083.317.25242.8047.11335.79416.70417.10333.94118.08152.82448.947
Learning Based [13]20092.818.97345.1457.45434.62116.32318.09436.77319.50348.00139.161
Nonlocal mattings [5]20119.329.678114.94914.849121.991131.03933.66961.341037.048117.021193.189
Shared Matting [4]20105.924.21568.2579.00649.95721.88724.25742.74737.14951.50239.512
Comprehensive Samplings [10]20084.926.80735.36115.591034.80215.73118.29539.17627.81555.93648.736
KNN Matting [1]20128.538.859120.081026.541144.01628.25831.83857.58935.78785.37975.348
Spectral Mattings [6]20129.272.4511155.21115.031104.95944.501135.841064.961166.661085.238130.3210
Sparse Samplings [14]20163.723.66439.99311.54835.17316.17216.48138.93527.46451.93344.554
Bayesian Matting [2]200110.368.3210110.78829.7011119.541047.661052.241167.6011118.011198.3410148.1711
Robust Matting [12]20076.032.72753.1259.85556.83825.40727.41642.73436.49662.55752.275
Refine Edge [15]20132.915.82157.5766.58240.25520.67518.04137.30219.11155.24344.183
Closed Form [7]20083.118.75248.3037.76336.78319.57418.21335.50119.15257.13453.956
Learning Based [13]20092.721.35350.9147.84435.59118.92219.37439.75321.31351.66141.992
Nonlocal mattings [5]20119.131.256129.13917.969123.491131.74936.131062.32938.828127.061191.379
Shared Matting [4]20106.027.95593.58710.60655.32725.04627.72748.28765.06958.63541.341
Comprehensive Samplings [10]20085.234.62842.66216.53835.69219.23321.26544.33633.28560.01655.617
KNN Matting [1]20128.242.719130.141026.501043.95629.19833.06859.22838.04782.47876.168
Spectral Mattings [6]20129.181.1411183.76115.491108.67954.311135.81967.311070.451082.749128.4110
Sparse Samplings [14]20163.427.39441.55112.15736.95417.04118.19243.94532.67454.81249.264
Bayesian Matting [2]20019.11.8593.5880.12813.3390.7492.31111.4784.51108.01916.2310
Robust Matting [12]20075.40.8970.9410.0724.8280.2550.8860.7741.5873.8973.927
Refine Edge [15]20134.30.5531.4860.0613.9960.3460.5450.7030.8133.8063.164
Closed Form [7]20082.40.4911.3540.0952.8810.1920.3520.6110.6112.7523.385
Learning Based [13]20092.30.5021.4250.0962.8820.1810.3410.6820.6422.3712.431
Nonlocal mattings [5]20119.91.5688.17100.701020.97111.11101.88102.51112.35911.171113.379
Shared Matting [4]20105.70.7962.7370.0944.5070.3571.0070.9171.4362.9432.943
Comprehensive Samplings [10]20084.90.7451.0620.1993.1540.2230.3940.8661.2453.6553.406
KNN Matting [1]20128.42.28106.4091.22113.4750.6181.0281.7391.9486.4287.738
Spectral Mattings [6]20129.77.031113.25110.08315.36102.09111.8492.45106.59118.711019.9711
Sparse Samplings [14]20163.90.6641.3030.1173.0930.2540.3830.8651.0442.9442.762
Bayesian Matting [2]20019.93.12105.0780.22816.65101.34103.03112.50911.601118.031125.6311
Robust Matting [12]20076.41.3471.2630.1556.1980.5071.4270.9962.3875.2175.447
Refine Edge [15]20133.70.6621.6260.0813.9560.4250.6350.7620.9234.0553.162
Closed Form [7]20082.60.6111.4640.1133.2420.2740.4530.6910.8013.2923.935
Learning Based [13]20092.30.6631.5950.1143.2210.2620.4210.7830.8422.9612.871
Nonlocal mattings [5]20119.41.6288.93100.781021.62111.2292.0692.77102.50813.521012.309
Shared Matting [4]20106.31.0353.5170.1765.7770.4861.5181.6272.5794.0443.784
Comprehensive Samplings [10]20084.91.0960.9510.2693.6350.2730.4840.9841.7154.6564.426
KNN Matting [1]20127.63.0596.8391.32113.4930.7081.1161.9682.2866.3088.558
Spectral Mattings [6]20129.49.361120.54110.11216.5593.15112.36102.91117.89108.87922.4410
Sparse Samplings [14]20163.50.7941.1820.1873.5240.2610.4420.9851.2943.3933.363
Bayesian Matting [2]200110.56.711011.0993.971119.88103.00105.18114.081137.451116.501125.0811
Robust Matting [12]20077.32.4082.5460.2357.6581.1882.7091.5764.1687.2687.157
Refine Edge [15]20132.80.8112.1850.1213.9140.5450.8450.9021.1414.5133.391
Closed Form [7]20083.00.8821.7030.2463.6430.4630.7230.8511.2024.3625.215
Learning Based [13]20092.41.0331.8840.2133.6220.4320.6111.0031.3234.2713.742
Nonlocal mattings [5]20118.51.76611.85101.10922.25111.2992.4673.0092.88516.361012.209
Shared Matting [4]20106.61.7456.0270.2777.4670.8562.7082.2677.6596.3565.054
Comprehensive Samplings [10]20085.32.2871.5520.3283.9750.5440.7341.3543.0166.5976.116
KNN Matting [1]20126.94.2497.8481.32103.5210.8871.3862.3483.0476.1759.878
Spectral Mattings [6]20129.313.171131.38110.18218.1695.33112.78103.741010.24109.19922.9210
Sparse Samplings [14]20163.41.3741.3210.2344.0260.3510.6521.4352.1944.6744.663
  1. city
  2. rain
  3. concert
  4. flowers
  5. snow
  6. Slava
  7. Vitaliy
  8. Artem
  9. juneau
  10. woods
  1. Source
  2. Trimap
  3. BM [2]
  4. RM [12]
  5. RE [15]
  6. CF [7]
  7. LB [13]
  8. NlM [5]
  9. ShM [4]
  10. CS [10]
  11. KNN [1]
  12. SpM [6]
  13. SpSM [14]
0 %
 
Note: Make sure you are using the latest version of your web browser (we recommend to use chromium-based web browsers)

Integral Plots

Multidimensional Analysis

References

[1]Qifeng Chen, Dingzeyu Li, and Chi-Ke­ung Tang. KNN mat­ting. Trans­ac­tions on Pat­tern Analy­sis and Ma­chine In­tel­li­gence (TPAMI), 35(9):2175–2188, 2013. [ doi ,  pro­ject page ]
[2]Yung-Yu Chuang, Brian Cur­less, David H. Salesin, and Richard Szeliski. A bayesian ap­proach to dig­i­tal mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), vol­ume 2, pages II-264–II-271, 2001. [ doi ,  pro­ject page ,  code ]
[3]Mikhail Ero­feev, Yury Git­man, Dmitriy Va­tolin, Alexey Fe­dorov, Jue Wang. Per­cep­tu­ally Mo­ti­vated Bench­mark for Video Mat­ting. British Ma­chine Vi­sion Con­fer­ence (BMVC), pages 99.1–99.12, 2015. [ doi ,  pdf   pro­ject page ]
[4]Ed­uardo S.L. Gastal and Manuel M. Oliveira. Shared sam­pling for real-time al­pha mat­ting. Com­puter Graph­ics Fo­rum, 29(2):575–584, 2010. [ pro­ject page ]
[5]Philip Lee and Ying Wu. Non­lo­cal mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 2193–2200, 2011. [ code ]
[6]A. Levin, A. Rav Acha, and D. Lischin­ski. Spec­tral mat­ting. Trans­ac­tions on Pat­tern Analy­sis and Ma­chine In­tel­li­gence (TPAMI), 30(10):1699–1712, 2008. [ doi ,  pro­ject page ]
[7]Anat Levin, Dani Lischin­ski, and Yair Weiss. A closed-form so­lu­tion to nat­ural im­age mat­ting. Trans­ac­tions on Pat­tern Analy­sis and Ma­chine In­tel­li­gence (TPAMI), 30(2):228–242, 2008. [ doi ,  code ]
[8]Christoph Rhe­mann, Carsten Rother, Jue Wang, Margrit Gelautz, Push­meet Kohli, and Pamela Rott. A per­cep­tu­ally mo­ti­vated on­line bench­mark for im­age mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 1826–1833, 2009. [ doi ]
[9]E. Shahrian and D. Ra­jan. Weighted color and tex­ture sam­ple se­lec­tion for im­age mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 718–725, 2012. [ doi ,  code ]
[10]E. Shahrian, D. Ra­jan, B. Price, and S. Co­hen. Im­prov­ing im­age mat­ting us­ing com­pre­hen­sive sam­pling sets. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 636–643, 2013. [ doi ,  code ]
[11]Karen Symonyan, Sergey Gr­ishin, Dmitriy Va­tolin, and Dmitriy Popov. Fast video su­per­res­o­lu­tion via clas­si­fi­ca­tion. In­ter­na­tional Con­fer­ence on Im­age Pro­cess­ing (ICIP), pages 349–352, 2008. [ doi ]
[12]Jue Wang and Michael F. Co­hen. Op­ti­mized color sam­pling for ro­bust mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 1–8, 2007. [ doi ,  pro­ject page ]
[13]Yuan­jie Zheng and C. Kamb­hamettu. Learn­ing based dig­i­tal mat­ting. In In­ter­na­tional Con­fer­ence on Com­puter Vi­sion (ICCV), pages 889–896, 2009. [ doi ,  code ]
[14]Sparse Sam­pling Mat­ting. Anony­mous TIP sub­mis­sion.
[15]http://​www.adobe.com/​en/​prod­ucts/​af­ter­ef­fects.html, Re­fine Edge tool in Adobe Af­ter Ef­fects CC.
Up