Background: To bind a Xorg window content to a OpenGL FBConfig, which
has to match the color format the Xorg window is using.
Previously, compton just find a FBConfig that has the same depth. This
led to chjj/compton#477, which has been fixed by a ugly hack.
The commit refactor the lookup mechanism to take as much into
consideration as we reasonably can. Hopefully preventing similar
breakages in the future.
Also, some code sharing between the old and new glx backend.
Signed-off-by: Yuxuan Shui <yshuiv7@gmail.com>