You could use the wildcard and pipe the results to Remove-Item.
dir \\server1\c$\users\*\downloads\* | where LastWriteTime -lt $cutoff | rm -Confirm
In this example, I removed files. But you could also remove whole directories. You will want to use the -Recurse and -Force switches.