This paper introduces a new simulation technique to enable detailed dexterous manipulation of cloth. Without reimplementation or sub- stantial modification, existing cloth simulators can only be used to approximate limited interaction between cloth and rigid bodies due to the incorrect computation of contact forces. For example, a sim- ple scenario of two fingers pinching a piece of cloth often results in the cloth slipping out of the hand. Our technique provides a simple solution to cloth-rigid coupling using existing cloth and rigid body simulators as-is. We develop a light-weight interface so that the rigid body and cloth simulators communicate on a demand-driven manner to achieve two main goals: allow the rigid bodies to impart friction forces to the cloth and avoid unsolvable collision situations between the rigid bodies and the cloth. We demonstrate a set of basic manipulation skills including gripping, pinching, and press- ing, that are frequently seen in daily activities such as dressing and folding clothes.