Disable source port rewriting - by default, PFSense rewrites the source port on all outbound traffic. Rewriting the source port of RTP can cause one way audio. In that case, you want to use manual outbound NAT and Static Port on all UDP traffic potentially with the exclusion of UDP 5060.
Set Conservative state table optimization - PFSense's default UDP timeouts are too low. Set "Firewall Optimization Options" to Conservative under System -> Advanced.