A high quality magnum pellet rifle like the patriot or falcon will shoot a .25 with about 26 to 32 fpe which is about as much as a a .22 short. it takes about 6 fpe to kill a tree rat and about 18 fpe to kill a coon effectivly so as long as you configure the fpe of your air rifle and it is high enough it will work. Here is a caculator
http://www.airguns.net/calculators.php
this will tell you what your fpe is as long as you know what the pellet size is and the fps.
Keep in mind most .177 pellets are on the average of 7.5 grains so that pellet will most likely not be effective. unless its shooting 1200fps Now with that being said you can actually kill a coon with only 9fpe if the pellet placement is correct but at night while trying to hold the light and the coon up in the tree , I would opt for much more than that. 18 to 20 fpe is about what it will take to really knock the coon out of a tree with ease.
So the smallest caliber I would use would be the .22 air rifle but the .25 will be much more effective.
There are some pcp air rifles that will shoot 45 fpe in .22 caliber with ease. So using the right air rifle is the key.