„Host access to custom network“ insecure?

Hey, can someone tell me, if there are any downsides of enabling „host access to custom networks“?

I have nextcloud aio and immich setup with custom networks and every other container on bridge. Accept for adgurd, which is on br0.

I want to install a proxy for all my containers, including immich and nextcloud, which only seems possible if I enable „host access to custom networks“.

Or do I miss something? Thx in advance!