...
-Organic Groups allow users to creates groups and sub groups but with no hierarchy. http://drupal.org/project/og
-Actually, this one is even better_: http://drupal.org/project/nodehierarchy
Other Considerations
- There is this project already which may provide the needed functionality. http://drupal.org/project/relativity
- Module Sub Groups
-The reason for the menu depth limitation seems to be documented here: http://drupal.org/node/274270
-It seems that both the Book module and therefore subgroups conform to the MENU_MAX_DEPTH global variable.
-It appears to be non-trivial to remove this limitation or even make it bigger (i.e. allow deeper menus/books/subgroups).
New Module: TreeGroups
-Creating our own module to solve this problem (TreeGroups) is probably the best path to take.
...