Jason Brennan explains brilliantly:
The libertarian responds by saying, “Doesn’t the IDA undermine left-liberalism as well, not just libertarianism?” They respond by saying, “Oh, no, because civil liberties, unlike economic liberties, are special.” But this just means that the IDA is irrelevant to determining whether a government action violates our rightful liberties or holdings.