Nginx reverse proxy and TLS configuration

Hi there. Probably I'm asking too many things in just one question, but I hope someone will be able to help me.

Recently I started using nginx as reverse proxy for my house so that I can configure subdomains to access different things.

I currently have one raspberry running a docker container with nginx, a server with proxmox and a fritzbox router.

I'm learning configurations little by little, but I figured out how to configure all the reverse proxy routes (even if some still give me some troubles). What I'm trying to do now is to configure tls for the 3 machines.

Proxmox was the easiest because the certificates were already present on the server, but now I'm trying to configure the router which has a ".cer" certificate. I read a post about it telling that as long as it respects some kind of format it would have been ok to use, but I didn't figure out how to put it inside the configuration.

I will also need to configure certificates for the containers inside my proxmox machine, so how will I create them?

Last question for this monolithic post is: Once I configure the certificates for the 3 machines, how can I make "my connection secure"? As I already said I configured the route for proxmox with the certificates too, but whenever I connect i always get the connection not secure message.

This is the configuration I made for proxmox

upstream proxmox {
server 192.168.x.x:8006;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name proxmox.mydomain.com;
ssl_certificate /etc/ssl/private/server/pve-ssl.pem;
ssl_certificate_key /etc/ssl/private/server/pve-ssl.key;
location / {
proxy_pass https://proxmox;
proxy_buffering off;
client_max_body_size 0;
proxy_connect_timeout 3600s;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
send_timeout 3600s;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_no_cache 1;
proxy_cache_bypass 1;
}
}
server {
listen 80;
listen [::]:80;
server_name proxmox.mydomain.com;
return 301 https://proxmox.mydomain.com;
}