Refactor FBConfig lookup

Background: To bind a Xorg window content to a OpenGL FBConfig, which
has to match the color format the Xorg window is using.

Previously, compton just find a FBConfig that has the same depth. This
led to chjj/compton#477, which has been fixed by a ugly hack.

The commit refactor the lookup mechanism to take as much into
consideration as we reasonably can. Hopefully preventing similar
breakages in the future.

Also, some code sharing between the old and new glx backend.

Signed-off-by: Yuxuan Shui <yshuiv7@gmail.com>

src/utils.h

@@ -73,6 +73,10 @@ static inline long attr_const min_l(long a, long b) {
 	return (a > b ? b : a);
 }
 
+static inline int attr_const popcountl(unsigned long a) {
+	return __builtin_popcountl(a);
+}
+
 /**
  * Normalize a double value to a specific range.
  *