What’s new

16.11.16We published evaluation results for Deep Matting [16].
6.04.16We published evaluation results for Sparse Sampling Matting [14].
15.12.151) New sequences with natural hairs including 3 public sequences.
2) New metrics of temporal coherency chosen by careful analysis (see [3]).
3) New trimap generation method (more natural-looking and accurate).
4) Better ground-truth quality owing to correction of lightning changes during capturing.
5) Improvement in website loading speed and interface.
7.09.15We published the paper with our benchmark description [3].
30.12.14We published results for multiple levels of trimaps; use drop-down menu at the top left corner to switch levels.
29.12.14We added general ranking to the rating table.
10.11.14“Sparse codes as Alpha Matte” was added.
26.09.14Source sequences are available for online view now. Full screen mode was added.
30.08.14Composite sequences are available now.
27.08.14“Refine edge tool in Adobe After Effects” was added.
25.08.14The official opening.

Overview

Introduction

The Video­Mat­ting pro­ject is the first pub­lic ob­jec­tive bench­mark for video-mat­ting meth­ods. It con­tains scat­ter plots and rat­ing ta­bles for dif­fer­ent qual­ity met­rics. In ad­di­tion, re­sults for par­tic­i­pat­ing meth­ods are avail­able for view­ing on a player equipped with a mov­able zoom re­gion. We be­lieve our work will help rank ex­ist­ing meth­ods and aid de­vel­op­ers of new meth­ods in im­prov­ing their re­sults.

Datasets

The data set con­sists of five mov­ing ob­jects cap­tured in front of a green plate and seven cap­tured us­ing the stop-mo­tion pro­ce­dure de­scribed be­low. We com­posed the ob­jects over a set of back­ground videos with var­i­ous lev­els of 3D cam­era mo­tion, color bal­ance, and noise. We pub­lished ground-truth data for two stop-mo­tion se­quences and hid the rest to en­sure fair­ness of the com­par­i­son.

Us­ing thresh­old­ing and mor­pho­log­i­cal op­er­a­tions on ground-truth al­pha mattes, we gen­er­ated nar­row trimaps. Then, we di­lated the re­sults us­ing graph­cut-based en­ergy min­i­miza­tion which pro­vides us with more hand­made-look­ing trimaps than com­mon mor­pho­log­i­cal di­la­tion.

Chroma Keying

green screen
stop motion
Alpha mattes from chroma keying and stop-motion capture for the same image region. The stop-motion result is significantly better at preserving details.

Chroma key­ing is a com­mon prac­tice of the cin­ema in­dus­try: the cin­e­matog­ra­pher cap­tures an ac­tor in front of a green or blue screen, then the VFX ex­pert re­places the back­ground us­ing spe­cial soft­ware. Our eval­u­a­tion uses five green-screen video se­quences with a sig­nif­i­cant amount of semi­trans­parency (e.g., hair or mo­tion blur), pro­vided to us by Hol­ly­wood cam­era work. We ex­tract al­pha mattes and cor­re­spond­ing fore­grounds us­ing The Foundry Key­light. Chroma key­ing en­ables us to get al­pha mattes of nat­ural-look­ing ob­jects with ar­bi­trary mo­tion. Nev­er­the­less, this tech­nique can’t guar­an­tee that the al­pha maps are nat­ural, be­cause it as­sumes the screen color is ab­sent from the fore­ground ob­ject. To get al­pha maps that have a very nat­ural ap­pear­ance, we use the stop-mo­tion method.

Stop Motion

One-step capture over different backgrounds. We use checkerboard backgrounds instead of solid ones to eliminate screen reflection.

We de­signed the fol­low­ing pro­ce­dure to per­form stop-mo­tion cap­ture: A fuzzy toy is placed on the plat­form in front of an LCD mon­i­tor. The toy ro­tates in small, dis­crete steps along a pre­de­fined 3D tra­jec­tory, con­trolled by two ser­vos con­nected to a com­puter. Af­ter each step the dig­i­tal cam­era in front of the setup cap­tures the mo­tion­less toy against a set of back­ground im­ages. At the end of this process, the toy is re­moved and the cam­era again cap­tures all of the back­ground im­ages.

We paid spe­cial at­ten­tion to avoid­ing re­flec­tions of the back­ground screen in the fore­ground ob­ject. These re­flec­tions can lead to false trans­parency that is es­pe­cially no­tice­able in non­trans­par­ent re­gions. To re­duce the amount of re­flec­tion we used checker­board back­ground im­ages in­stead of solid col­ors, thereby ad­just­ing the mean color of the screen to be the same for each back­ground.

At the end we cor­rected global light­ing changes caused by light bulb flick­er­ing. Thus fi­nally we ob­tain al­pha mattes with less than 1% of noise level. The de­tailed de­scrip­tion of ground-truth ex­trac­tion meth­ods is given in [3].

Evaluation Methodology

Our com­par­i­son in­cludes both im­age- and video-mat­ting meth­ods. We ap­ply each mat­ting method to the videos in our data set, and then com­pare the re­sults us­ing the fol­low­ing met­rics of per-pixel ac­cu­racy and tem­po­ral co­herency (look into our pa­per [3] for com­par­i­son of dif­fer­ent met­rics):

Equationmultiline equation
Equationmultiline equation
Equationmultiline equation

Here Equationnumber-sign de­notes to­tal num­ber of pix­els, Equationalpha Subscript p comma t and EquationModifyingAbove alpha With caret Subscript p comma t de­note trans­parency val­ues of video mat­ting un­der con­sid­er­a­tion and ground truth cor­re­spond­ingly at pixel Equationp of frame Equationt, and Equationv Subscript p de­notes mo­tion vec­tor at pixel Equationp. We use op­ti­cal-flow al­go­rithm [11] com­puted for ground-truth se­quences. It is worth not­ing that mo­tion-aware met­rics will not give un­fair ad­van­tage to mat­ting meth­ods based on the sim­i­lar mo­tion es­ti­ma­tion method since they do not have ground truth se­quence. The de­tailed de­scrip­tion of used qual­ity met­rics is given in [3].

Public Sequences

For the train­ing pur­poses we pub­lish here three test se­quences with their ground-truth trans­parency maps. De­vel­op­ers and re­searchers are wel­come to use these se­quences, but we ask to cite us [3]

Participate

We in­vite de­vel­op­ers of video-mat­ting meth­ods to use our bench­mark. We will eval­u­ate the sub­mit­ted data and re­port scores to the de­vel­oper. In cases where the de­vel­oper specif­i­cally grants per­mis­sion, we will pub­lish the re­sults on our site. We can also pub­lish anony­mous scores for blind-re­viewed pa­pers. To par­tic­i­pate, sim­ply fol­low these steps:

  1. Download the data set containing our sequences: City, Flowers, Concert, Rain, Snow, Vitaliy, Artem, Slava, Juneau, Woods,
  2. Apply your method to each of our test cases
  3. Upload the alpha and foreground sequences to any file-sharing service. We kindly ask you to maintain these naming and directory-structure conventions. If your method doesn't explicitly produce the foreground images you can skip uploading them; in this case, we will generate them using method proposed in [7].
  4. Fill in this form to provide information about your method

Con­tact us by email with any ques­tions or sug­ges­tions at ques­tions@video­mat­ting.com.

Cite Us

To re­fer to our eval­u­a­tion or test se­quences in your work cite our pa­per [3].

                
@inproceedings{Erofeev2015,
	title={Perceptually Motivated Benchmark for Video Matting},
	author={Mikhail Erofeev and Yury Gitman and  Dmitriy Vatolin and Alexey Fedorov and Jue Wang},
	year={2015},
	month={September},
	pages={99.1-99.12},
	articleno={99},
	numpages={12},
	booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
	publisher={BMVA Press},
	doi={10.5244/C.29.99},
	isbn={1-901725-53-7},
	url={https://dx.doi.org/10.5244/C.29.99}
}
                

Evaluation

Rating

Trimap size:
Trimap available for each frame
year
rank
city
rain
concert
flowers
snow
Slava
Vitaliy
Artem
juneau
woods
Bayesian Matting [2]20018.858.761051.66910.795144.53930.36953.28943.70970.501096.698140.1210
Robust Matting [12]20075.337.54726.1529.61288.15518.52533.31734.15446.91878.56780.416
Refine Edge [15]20136.737.78838.4878.641129.56727.47831.36636.65838.755102.02994.508
Closed Form [7]20084.731.34331.80410.967149.481015.86223.67430.15230.60369.94582.137
Learning Based [13]20093.229.98233.08510.846134.82815.23123.35231.46330.20158.38256.602
Nonlocal mattings [5]201110.754.21992.371133.8611243.471245.131164.051161.751167.309135.0811171.7911
Shared Matting [4]20105.836.79646.72810.30487.64424.41735.18835.26645.23768.52467.174
Comprehensive Samplings [10]20084.834.48530.18312.10879.60216.17325.02535.25541.05677.72670.185
KNN Matting [1]20129.967.831192.221064.7312112.46632.961061.501050.871079.2011113.4110123.639
Spectral Mattings [6]201211.686.3212105.241213.159210.041148.761265.721268.8312109.4412157.1812182.8212
Sparse Samplings [14]20164.033.42433.5069.94381.06318.42423.64335.83738.00465.17364.123
Deep Matting [16]20162.527.33121.40120.331044.93119.23619.99130.00130.55246.18146.391
Bayesian Matting [2]20019.873.181067.92916.069157.281039.591060.37956.051085.5311115.8810169.3310
Robust Matting [12]20076.548.19832.89414.72799.81525.96642.51738.88659.08893.79796.137
Refine Edge [15]20136.442.23741.74710.451127.02731.40835.08639.46742.915108.71897.768
Closed Form [7]20084.634.53435.86513.354154.12921.64527.19433.01235.45381.48492.756
Learning Based [13]20093.633.44337.63613.233138.31819.95426.51334.51334.70268.41263.532
Nonlocal mattings [5]201110.555.04997.851031.8110242.751246.941167.111165.091169.019144.6611176.6111
Shared Matting [4]20106.341.44656.75814.14596.71427.66742.92840.05858.17783.02578.915
Comprehensive Samplings [10]20084.238.43528.29214.45683.85216.84227.29537.07447.38689.96678.454
KNN Matting [1]20129.679.351198.711166.2412112.81634.60964.171054.44984.3810112.139128.869
Spectral Mattings [6]201211.599.0512125.291216.048214.421159.611274.551277.2512122.8912162.7312188.1612
Sparse Samplings [14]20163.033.20231.92312.34285.52317.66324.64237.78539.87471.05369.943
Deep Matting [16]20162.028.67121.10141.801144.81116.57121.04130.57130.94146.40150.271
Bayesian Matting [2]200110.298.131195.95938.5410170.761059.511180.721167.6410122.0810135.5910179.2510
Robust Matting [12]20077.367.71949.02619.875109.79544.10962.18850.54781.498118.098115.968
Refine Edge [15]20136.150.31551.81715.852125.25740.73845.38646.43652.225118.889103.306
Closed Form [7]20085.045.04441.69424.197159.47930.09536.85539.20246.973104.074110.267
Learning Based [13]20093.944.97343.75521.906143.54826.50433.89340.44345.93290.35381.222
Nonlocal mattings [5]20119.956.767113.901133.979241.311250.741077.141068.371176.437159.0811181.1411
Shared Matting [4]20106.455.84679.09819.784107.45432.94660.78751.478122.3311107.34597.225
Comprehensive Samplings [10]20084.356.79837.36316.96387.64224.04334.32443.69464.636110.50693.354
KNN Matting [1]20128.895.8910112.701066.1212113.10637.18771.26960.03998.429110.877138.669
Spectral Mattings [6]201211.5116.8712157.371226.128218.941176.221288.741291.6912148.7012169.3312193.4612
Sparse Samplings [14]20162.643.32234.66214.78189.79319.23229.92244.59551.94485.44281.773
Deep Matting [16]20162.031.29123.16153.021144.93118.20124.36132.04137.60150.48154.761
Bayesian Matting [2]200110.035.931065.2499.287104.681127.61935.041246.41960.341188.4611121.9111
Robust Matting [12]20075.222.32632.7327.26347.27916.04518.72734.47424.96553.42742.404
Refine Edge [15]20134.614.41144.0866.30240.69618.36716.53334.93517.83253.43843.606
Closed Form [7]20083.416.37241.2947.67435.14514.59216.40232.43217.37151.40447.718
Learning Based [13]20093.517.29442.9957.88634.03314.44117.07534.37318.37346.02239.463
Nonlocal mattings [5]201110.629.079110.761014.1011120.011229.821132.691059.911136.2210108.431296.8210
Shared Matting [4]20106.522.72759.5987.72545.92820.34822.35838.02829.56847.45338.192
Comprehensive Samplings [10]20085.722.78839.97313.831033.11215.61317.51637.09625.72653.33645.047
KNN Matting [1]20129.836.1611115.081125.581244.09727.801031.61956.081034.80987.531075.369
Spectral Mattings [6]201210.365.5012131.74124.781103.611039.031234.391163.211264.401287.449122.5312
Sparse Samplings [14]20165.821.78544.8579.86834.07416.94616.55437.40725.84751.77542.985
Deep Matting [16]20162.616.79325.93113.12926.74115.64414.58131.85119.25445.03137.271
Bayesian Matting [2]200110.546.961177.27911.327111.601133.851139.541256.86975.5012102.7011151.6012
Robust Matting [12]20076.326.36737.7338.35552.07919.18621.45737.08528.55757.78847.956
Refine Edge [15]20134.115.24147.3576.58240.54619.40716.77335.74318.24254.11643.434
Closed Form [7]20084.117.25342.8057.11335.79516.70517.10433.94218.08152.82548.948
Learning Based [13]20093.718.97445.1467.45434.62216.32418.09536.77419.50448.00239.162
Nonlocal mattings [5]201110.229.679114.941014.849121.991231.031033.661061.341137.049117.021293.1810
Shared Matting [4]20106.824.21668.2589.00649.95821.88824.25842.74837.141051.50339.513
Comprehensive Samplings [10]20085.826.80835.36215.591034.80315.73218.29639.17727.81655.93748.737
KNN Matting [1]20129.538.8510120.081126.541244.01728.25931.83957.581035.78885.371075.349
Spectral Mattings [6]201210.172.4512155.21125.031104.951044.501235.841164.961266.661185.239130.3211
Sparse Samplings [14]20164.623.66539.99411.54835.17416.17316.48238.93627.46551.93444.555
Deep Matting [16]20162.317.21224.74121.031127.68114.33114.98131.41119.05345.10136.961
Bayesian Matting [2]200111.368.3211110.78929.7012119.541147.661152.241267.6012118.011298.3411148.1712
Robust Matting [12]20076.932.72853.1269.85556.83925.40827.41742.73536.49762.55852.276
Refine Edge [15]20133.615.82157.5776.58240.25620.67618.04237.30319.11155.24444.184
Closed Form [7]20083.918.75348.3047.76336.78419.57518.21435.50219.15257.13553.957
Learning Based [13]20093.621.35450.9157.84435.59218.92319.37539.75421.31451.66241.993
Nonlocal mattings [5]201110.031.257129.131017.969123.491231.741036.131162.321038.829127.061291.3710
Shared Matting [4]20106.927.95693.58810.60655.32825.04727.72848.28865.061058.63641.342
Comprehensive Samplings [10]20086.134.62942.66316.53835.69319.23421.26644.33733.28660.01755.618
KNN Matting [1]20129.242.7110130.141126.501143.95729.19933.06959.22938.04882.47976.169
Spectral Mattings [6]201210.081.1412183.76125.491108.671054.311235.811067.311170.451182.7410128.4111
Sparse Samplings [14]20164.327.39541.55212.15736.95517.04218.19343.94632.67554.81349.265
Deep Matting [16]20162.218.72228.24123.651028.18115.36116.66131.80120.98347.83136.951
Bayesian Matting [2]200110.01.85103.5890.12813.33100.74102.31121.4794.51118.011016.2311
Robust Matting [12]20076.30.8980.9420.0724.8290.2560.8870.7751.5883.8983.928
Refine Edge [15]20135.20.5541.4870.0613.9970.3470.5460.7040.8143.8073.165
Closed Form [7]20083.00.4911.3550.0952.8820.1920.3530.6110.6122.7533.386
Learning Based [13]20093.10.5031.4260.0962.8830.1810.3420.6830.6432.3722.432
Nonlocal mattings [5]201110.91.5698.17110.701120.97121.11111.88112.51122.351011.171213.3710
Shared Matting [4]20106.60.7972.7380.0944.5080.3581.0080.9181.4372.9442.944
Comprehensive Samplings [10]20085.80.7461.0630.1993.1550.2240.3950.8671.2463.6563.407
KNN Matting [1]20129.42.28116.40101.22123.4760.6191.0291.73101.9496.4297.739
Spectral Mattings [6]201210.67.031213.25120.08315.36112.09121.84102.45116.59128.711119.9712
Sparse Samplings [14]20164.80.6651.3040.1173.0940.2550.3840.8661.0452.9452.763
Deep Matting [16]20162.30.5020.4810.32101.1510.2230.2510.6320.5711.8411.841
Bayesian Matting [2]200110.83.12115.0790.22816.65111.34113.03122.501011.601218.031225.6312
Robust Matting [12]20077.31.3481.2640.1556.1990.5081.4280.9972.3885.2185.448
Refine Edge [15]20134.60.6631.6270.0813.9570.4260.6360.7630.9244.0563.163
Closed Form [7]20083.50.6121.4650.1133.2430.2750.4540.6920.8023.2933.936
Learning Based [13]20093.20.6641.5960.1143.2220.2630.4220.7840.8432.9622.872
Nonlocal mattings [5]201110.31.6298.93110.781021.62121.22102.06102.77112.50913.521112.3010
Shared Matting [4]20107.21.0363.5180.1765.7780.4871.5191.6282.57104.0453.785
Comprehensive Samplings [10]20085.81.0970.9520.2693.6360.2740.4850.9851.7164.6574.427
KNN Matting [1]20128.63.05106.83101.32123.4940.7091.1171.9692.2876.3098.559
Spectral Mattings [6]201210.39.361220.54120.11216.55103.15122.36112.91127.89118.871022.4411
Sparse Samplings [14]20164.40.7951.1830.1873.5250.2620.4430.9861.2953.3943.364
Deep Matting [16]20162.00.5610.4711.05111.1810.1910.2710.6710.5711.9011.991
Bayesian Matting [2]200111.56.711111.09103.971219.88113.00115.18124.081237.451216.501225.0812
Robust Matting [12]20078.22.4092.5470.2357.6591.1892.70101.5774.1697.2697.158
Refine Edge [15]20133.70.8122.1860.1213.9150.5460.8460.9031.1424.5143.392
Closed Form [7]20083.90.8831.7040.2463.6440.4640.7240.8521.2034.3635.216
Learning Based [13]20093.31.0341.8850.2133.6230.4330.6121.0041.3244.2723.743
Nonlocal mattings [5]20119.41.76711.85111.10922.25121.29102.4683.00102.88616.361112.2010
Shared Matting [4]20107.51.7466.0280.2777.4680.8572.7092.2687.65106.3575.055
Comprehensive Samplings [10]20086.22.2881.5530.3283.9760.5450.7351.3553.0176.5986.117
KNN Matting [1]20127.84.24107.8491.32103.5220.8881.3872.3493.0486.1769.879
Spectral Mattings [6]201210.213.171231.38120.18218.16105.33122.78113.741110.24119.191022.9211
Sparse Samplings [14]20164.31.3751.3220.2344.0270.3520.6531.4362.1954.6754.664
Deep Matting [16]20162.00.6410.6011.34111.2210.2210.3710.7510.7812.2712.211
  1. city
  2. rain
  3. concert
  4. flowers
  5. snow
  6. Slava
  7. Vitaliy
  8. Artem
  9. juneau
  10. woods
  1. Source
  2. Trimap
  3. BM [2]
  4. RM [12]
  5. RE [15]
  6. CF [7]
  7. LB [13]
  8. NlM [5]
  9. ShM [4]
  10. CS [10]
  11. KNN [1]
  12. SpM [6]
  13. SpSM [14]
  14. DM [16]
0 %
 
Note: Make sure you are using the latest version of your web browser (we recommend to use chromium-based web browsers)

Integral Plots

Multidimensional Analysis

References

[1]Qifeng Chen, Dingzeyu Li, and Chi-Ke­ung Tang. KNN mat­ting. Trans­ac­tions on Pat­tern Analy­sis and Ma­chine In­tel­li­gence (TPAMI), 35(9):2175–2188, 2013. [ doi ,  pro­ject page ]
[2]Yung-Yu Chuang, Brian Cur­less, David H. Salesin, and Richard Szeliski. A bayesian ap­proach to dig­i­tal mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), vol­ume 2, pages II-264–II-271, 2001. [ doi ,  pro­ject page ,  code ]
[3]Mikhail Ero­feev, Yury Git­man, Dmitriy Va­tolin, Alexey Fe­dorov, Jue Wang. Per­cep­tu­ally Mo­ti­vated Bench­mark for Video Mat­ting. British Ma­chine Vi­sion Con­fer­ence (BMVC), pages 99.1–99.12, 2015. [ doi ,  pdf   pro­ject page ]
[4]Ed­uardo S.L. Gastal and Manuel M. Oliveira. Shared sam­pling for real-time al­pha mat­ting. Com­puter Graph­ics Fo­rum, 29(2):575–584, 2010. [ pro­ject page ]
[5]Philip Lee and Ying Wu. Non­lo­cal mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 2193–2200, 2011. [ code ]
[6]A. Levin, A. Rav Acha, and D. Lischin­ski. Spec­tral mat­ting. Trans­ac­tions on Pat­tern Analy­sis and Ma­chine In­tel­li­gence (TPAMI), 30(10):1699–1712, 2008. [ doi ,  pro­ject page ]
[7]Anat Levin, Dani Lischin­ski, and Yair Weiss. A closed-form so­lu­tion to nat­ural im­age mat­ting. Trans­ac­tions on Pat­tern Analy­sis and Ma­chine In­tel­li­gence (TPAMI), 30(2):228–242, 2008. [ doi ,  code ]
[8]Christoph Rhe­mann, Carsten Rother, Jue Wang, Margrit Gelautz, Push­meet Kohli, and Pamela Rott. A per­cep­tu­ally mo­ti­vated on­line bench­mark for im­age mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 1826–1833, 2009. [ doi ]
[9]E. Shahrian and D. Ra­jan. Weighted color and tex­ture sam­ple se­lec­tion for im­age mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 718–725, 2012. [ doi ,  code ]
[10]E. Shahrian, D. Ra­jan, B. Price, and S. Co­hen. Im­prov­ing im­age mat­ting us­ing com­pre­hen­sive sam­pling sets. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 636–643, 2013. [ doi ,  code ]
[11]Karen Symonyan, Sergey Gr­ishin, Dmitriy Va­tolin, and Dmitriy Popov. Fast video su­per­res­o­lu­tion via clas­si­fi­ca­tion. In­ter­na­tional Con­fer­ence on Im­age Pro­cess­ing (ICIP), pages 349–352, 2008. [ doi ]
[12]Jue Wang and Michael F. Co­hen. Op­ti­mized color sam­pling for ro­bust mat­ting. In Com­puter Vi­sion and Pat­tern Recog­ni­tion (CVPR), pages 1–8, 2007. [ doi ,  pro­ject page ]
[13]Yuan­jie Zheng and C. Kamb­hamettu. Learn­ing based dig­i­tal mat­ting. In In­ter­na­tional Con­fer­ence on Com­puter Vi­sion (ICCV), pages 889–896, 2009. [ doi ,  code ]
[14]Sparse Sam­pling Mat­ting. Anony­mous TIP sub­mis­sion.
[15]http://​www.adobe.com/​en/​prod­ucts/​af­ter­ef­fects.html, Re­fine Edge tool in Adobe Af­ter Ef­fects CC.
[16]Deep Mat­ting. Anony­mous CVPR 2017 sub­mis­sion.
Up