I’m not sure about exemption, but I know you can put as many dependents as you’d like on your W-4, reducing the amount withheld. Just like you said, this affects withholding only and has no bearing on the taxes you’ll owe in April.
It’s a good idea to adjust your withholdings so you break even in April. Big refunds seem nice until you realize you gave the government an interest free loan with money you could’ve put to work for you.