Mandatory only in a specific parameterset

Let's say I have some params like this:

[CmdletBinding()]
param (
[Parameter(Mandatory)]
[string]$Param1,
[Parameter(Mandatory)]
[string]$Param2,
[Parameter(ParameterSetName = 'Set1',
[switch]$Param3,
[Parameter(ParameterSetName = 'Set1',
Mandatory)]
[string]$Param4
)

I'm forced to set params $Param1 and $Param2 always which I understand. However, I want to make the $Param4 only mandatory if I've used the $Param3 switch param. Is this possible? I'm trying this and finding that if I use Mandatory on the $Param4 switch, it's enforcing this all the time regardless of which params I choose. Am I going to have to go to dynamic params for this?