Playit latency causing server crashes

I am hosting a vanilla Minecraft server for a few friends and myself and we have had issues with a lot of server lag and the server crashing when people are exploring new chunks. Based on logs from the server I don’t believe its a problem caused by my server hosting computer. Further when i look at the latency of connecting to the server though play.it vs connection via a port forwarded connection the playit connection. The playit connection is constantly at 400+ ms where as the port forwarded connection is at 5-10 ms. is there a good way to fix this or is this just a fact of using playit.

Server Info:
Main OS is proxmox with a docker VM hosting the playit tunnel, second VM running crafty to host the server.

I have not had any of the same issues listed above when connection to the same server on my local network