Yeah I think you're right that this would work OK but it's just not perfect. Typically when I think of something like the below. So you COULD:
Have it such that when you click on these containers, a Product is added to the cart. From there maybe the container goes away (removing the customer's ability to remove the product from the cart). This would allow the user to remove the product from the cart via the cart removal feature like you mentioned
This would work and be OK but we're lacking the ability to:
Have some mechanism where you click these containers on an off and have this control some temp cart that is waiting in standby for the final Add to Cart click that then adds all the goods to the cart at once.
This is a nice feature but we just don't have it. We do have something similar via our Bundle Builder