No network in toolbox container

[btreut]

I successfully created my first container as suggested in [solved] Running windows applications on eos flatpak error install but I cannot install anything inside the container. Apparently I don’t have network connectivity inside. Both dnf & yum complain “failed to synchronize cache for repo ‘fedora-modular’”. Running dnf -v hinted at a network problem. In the container a ping mirrors.fedoraproject.org fails, but outside it works.

Any help and or ideas are welcome …
eos-diagnostic-191004_122558_UTC 0200.txt (841.5 KB)

Diving a bit deeper, I realize that the bridge network is missing, which is referenced when I look at details of my container by podman inspect mycontainer.

[egrath]

Just out of curiosity, do you have network available when creating a new container with toolbox? Eventually something went terribly wrong when toolbox built the container.

If no, can you please try to create a container manually with podman?

podman run -it ubuntu:19.04

This will fetch the latest Ubuntu Docker image and create it with the default parameters, which also include a bridged network.

[btreut]

Just out of curiosity, do you have network available when creating a new
container with toolbox ?

yes, there was a fully functional network connection via WLAN. The
toolbox create command downloaded the fedora-29 image (500MB) during
creation.

[egrath]

can you please try the commands i wrote in my post and see if you have network inside the container?

[btreut]

podman run -it ubuntu:19.04

An apt update works inside it.

But even in a parallel window trying a ping www.lrz.de returns a socket error in the fedora-29 container, which I created some days ago with toolbox create --container BlackBox. I append the output (15.8 KB) of podman inspect BlackBox.

[egrath]

OK, it really seems that during the creation of the container “BlackBox” something went wrong and the network didn’t got assigned properly. As podman does not allow to change settings of containers after it’s creation (https://github.com/containers/libpod/issues/376) your best bet would be to simply delete this container and use a new one.

[btreut]

Hmm, I tried to create a new (unnamed) container yesterday, but that new container had exactly the same behavior. And I retried again now:

bernhard@endless:~$ toolbox create
Created container: fedora-toolbox-29
Enter with: toolbox enter
bernhard@endless:~$ toolbox enter
bernhard@toolbox:~$ sudo dnf update

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Fedora Modular 29 - x86_64                                                                 0.0  B/s |   0  B     00:00    
Failed to synchronize cache for repo 'fedora-modular'
Error: Failed to synchronize cache for repo 'fedora-modular'
bernhard@toolbox:~$ `

I guessed that I have to delete also the downloaded image, but even with a complete new download: exactly the same as above:

bernhard@endless:~$ toolbox rmi registry.fedoraproject.org/f29/fedora-toolbox:29
bernhard@endless:~$ toolbox list
bernhard@endless:~$ toolbox create
Image required to create toolbox container.
Download registry.fedoraproject.org/f29/fedora-toolbox:29 (500MB)? [y/N]: y
Created container: fedora-toolbox-29
Enter with: toolbox enter
bernhard@endless:~$ toolbox enter
bernhard@toolbox:~$ sudo dnf update

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Fedora Modular 29 - x86_64                                                                 0.0  B/s |   0  B     00:00    
Failed to synchronize cache for repo 'fedora-modular'
Error: Failed to synchronize cache for repo 'fedora-modular'

BTW: I would prefer to have a debian image, but I was not able to find any.

[btreut]

I started with endless 3.3.8 (see one of my older posts) and updated/upgraded as suggested by the OS. I don’t know if something is missing due to upgrades or something is wrong with toolbox. Therefore I dare to file an issue on github in containers/toolbox …

[btreut]

I deleted the container once again and used toolbox -v create with the following result:

$ toolbox -v create
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user bernhard
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 'podman system migrate' doesn't exist
toolbox: Fedora generational core is f29
toolbox: base image is fedora-toolbox:29
toolbox: container is fedora-toolbox-29
toolbox: failed to read property Listen from sssd-kcm.socket
toolbox: parsing value  of property Listen in sssd-kcm.socket
toolbox: checking if 'podman create' supports --dns=none and --no-hosts
toolbox: 'podman create' supports --dns=none and --no-hosts
toolbox: looking for image localhost/fedora-toolbox:29
toolbox: looking for image registry.fedoraproject.org/f29/fedora-toolbox:29
toolbox: base image fedora-toolbox:29 resolved to registry.fedoraproject.org/f29/fedora-toolbox:29
toolbox: checking if container fedora-toolbox-29 already exists
toolbox: /sysroot/home/bernhard canonicalized to /sysroot/home/bernhard
toolbox: checking if /home is a symbolic link to /var/home
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: trying to create container fedora-toolbox-29
Created container: fedora-toolbox-29
Enter with: toolbox enter

but I have no idea, if this failed to read property Listen from sssd-kcm.socket causes my problems. See also this issue, but I don’t really understand everything discussed there.

[egrath]

I don’t know the exact reason for the problem, but, if the Ubuntu Image works for you, why don’t you simply use this one? If interested, i can write a simple tutorial on how to set up a good usable Ubuntu container with Podman, which acts similary to the toolbox created Fedora one (Access to the X Server, 3D Acceleration, …)

[btreut]

If interested, i can write a simple tutorial on how to set up a good usable Ubuntu container with Podman, which acts similary to the toolbox created Fedora one (Access to the X Server, 3D Acceleration, …)

thanks in advance, I would really apreciate a tutorial. I find the podman docu a bit confusing and did not find a way to store the state.

[egrath]

Here you go:

[Tutorial] Running Ubuntu with Podman

Introduction What is this all about? Podman is a technology, which allows you to run an arbitrary linux operating system inside something called a “container”. A “container” can be seen as a operating system, running inside another operating system like an application. Glossar Podman: The technology and also the main command used here (podman) Container: A isolated operating system running as an application and sharing resources with the host system Image: An operating system image used to …

[andrunko]

Hi @btreut, I tried reproducing this issue in a 3.6.4 installation but it just worked for me, so not sure what is going on here.

That said, could you please check how do /etc/hosts and /etc/resolv.conf look inside your toolbox container? They should be links to /run/host/etc/hosts and /run/host/etc/resolv.conf respectively, and /etc from your host should be bind mounted as /run/host/etc inside your container as described here (This upstream issue also has some details on similar issues, but bear in mind we don’t use buildah in the version of toolbox included in 3.6.4).

An alternative would be to wait for 3.7.0 which got all container related packages updated, so there is a high chance this issue won’t happen there. Or use podman directly as suggested by @egrath.

As for ping not working, this is expected (and unfortunately it will also not work on 3.7.0, but it will be included in a future release) - see this upstream issue for details.

Also, the sssd-kcm.socket message above can be ignored as it should not be related to this issue (fwiw it also happens here).

[btreut]

They should be links to /run/host/etc/hosts and /run/host/etc/resolv.conf respectively,

/etc/hosts is correctly linked to /run/host/etc/hosts, but /etc/resolv.conf is a dead link to /run/host/etc/resolv.conf. Outside the toolbox container /etc/resolv.conf links to /var/run/NetworkManager/resolv.conf. I guess that this link is unreachable inside the container.
And just as info: I am connected to a wlan with DHCP.

[andrunko]

/etc/hosts is correctly linked to /run/host/etc/hosts , but /etc/resolv.conf is a dead link to /run/host/etc/resolv.conf . Outside the toolbox container /etc/resolv.conf links to /var/run/NetworkManager/resolv.conf . I guess that this link is unreachable inside the container.

Does it work if you manually update resolv.conf to match the host? Yes, looks like an issue with toolbox not able to access the link inside the container - may be worth an upstream ticket.

[btreut]

manually update resolv.conf

yes a cp /etc/resolv.conf ~ outside the container and sudo rm /etc/resolv.conf && sudo cp resolv.conf /etc/ inside the container solves the network issue. dnf update works flawlessly.
@andrunko would you mind to file it upstream

[andrunko]

yes a cp /etc/resolv.conf ~ outside the container and sudo rm /etc/resolv.conf && sudo cp resolv.conf /etc/ inside the container solves the network issue. dnf update works flawlessly.
@andrunko would you mind to file it upstream

Great to hear, at least we have a workaround . I will check with upstream, this may have been fixed already there, lets see.

[btreut]

at least we have a workaround

yes, but only until changing to a new wlan and changed network config.

Another workaround would be (outside the container):
sudo mv /etc/resolv.conf{,_link} && cp $(readlink /etc/resolv.conf_link) /etc/resolv.conf
with the same limitations.