I can't enter toolbox anymore

#1

I just updated endless and now I can’t enter toolbox anymore

Screenshot from 2021-04-19 18-34-09 (1)
Screenshot from 2021-04-19 18-34-09 (1)892×560 21.9 KB

Any help would be appreciated :pray: :rose: :pray: :rose:

#2

Can you please run:

toolbox --verbose enter --release 33

And post the output here

1 Like
#3

this is the output :

toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user nazanin
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if ‘podman system migrate’ exists
toolbox: migration not needed: 1.5.1 is unchanged
toolbox: Fedora generational core is f33
toolbox: base image is fedora-toolbox:33
toolbox: container is fedora-toolbox-33
toolbox: checking if container fedora-toolbox-33 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-33
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-33
Error: unable to start container “fedora-toolbox-33”: error reading container (probably exited) json message: EOF
toolbox: failed to start container fedora-toolbox-33

#4

OK, please now run the following command and upload the logfile generated:

podman stop fedora-toolbox-33
echo PODMAN START >> ~/podman-log.txt
podman start --log-level=debug fedora-toolbox-33 2>&1 | tee -a ~/podman-log.txt
echo PODMAN EXEC >> ~/podman-log.txt
podman exec --interactive --tty fedora-toolbox-33 /bin/true 2>&1 | tee -a ~/podman-log.txt
echo PODMAN CONTAINER INSPECT >> ~/podman-log.txt
podman container inspect fedora-toolbox-33 2>&1 | tee -a ~/podman-log.txt

The file is named podman-log.txt in your home directory and contains very detailled information which should help us finding the root cause of the issue.

1 Like
#5

this is the result , which i don’t i understand a thing btw :sweat_smile:

podman-log.txt (29.2 KB)

#6

OK, please now:

  1. Reboot your device
  2. Open a Terminal, then run:
journalctl -f -t conmon
  1. Open another Terminal, then run:
podman start --log-level=debug fedora-toolbox-33
  1. The last command will fail, but in the first Terminal there should be some output. Please copy paste all of it into this Thread.
1 Like
#7

I really appreciate you helping me with this :rose: :rose: :pray: :pray:

This is the first command result :

– Logs begin at Sat 2021-04-17 19:54:00 +0430. –
Apr 19 21:54:34 endless conmon[37211]: conmon 81f9cc22be4270c89d2b : failed to write to /proc/self/oom_score_adj: Permission denied
Apr 19 21:54:34 endless conmon[37212]: conmon 81f9cc22be4270c89d2b : attach sock path: /run/user/1000/libpod/tmp/socket/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/attach
Apr 19 21:54:34 endless conmon[37212]: conmon 81f9cc22be4270c89d2b : addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/attach}
Apr 19 21:54:34 endless conmon[37212]: conmon 81f9cc22be4270c89d2b : ctl fifo path: /sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/ctl
Apr 19 21:54:34 endless conmon[37212]: conmon 81f9cc22be4270c89d2b : Failed to mkfifo at /sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/ctl: File exists
Apr 19 21:56:50 endless conmon[37375]: conmon 81f9cc22be4270c89d2b : failed to write to /proc/self/oom_score_adj: Permission denied
Apr 19 21:56:50 endless conmon[37376]: conmon 81f9cc22be4270c89d2b : attach sock path: /run/user/1000/libpod/tmp/socket/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/attach
Apr 19 21:56:50 endless conmon[37376]: conmon 81f9cc22be4270c89d2b : addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/attach}
Apr 19 21:56:50 endless conmon[37376]: conmon 81f9cc22be4270c89d2b : ctl fifo path: /sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/ctl
Apr 19 21:56:50 endless conmon[37376]: conmon 81f9cc22be4270c89d2b : Failed to mkfifo at /sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/ctl: File exists
– Reboot –
Apr 20 18:48:37 endless conmon[2341]: conmon 81f9cc22be4270c89d2b : failed to write to /proc/self/oom_score_adj: Permission denied
Apr 20 18:48:37 endless conmon[2342]: conmon 81f9cc22be4270c89d2b : attach sock path: /run/user/1000/libpod/tmp/socket/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/attach
Apr 20 18:48:37 endless conmon[2342]: conmon 81f9cc22be4270c89d2b : addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/attach}
Apr 20 18:48:37 endless conmon[2342]: conmon 81f9cc22be4270c89d2b : ctl fifo path: /sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/ctl
Apr 20 18:48:37 endless conmon[2342]: conmon 81f9cc22be4270c89d2b : terminal_ctrl_fd: 13
Apr 20 18:48:37 endless conmon[2342]: conmon 81f9cc22be4270c89d2b : Failed to create container: exit status 1

And this is for the second one :
WARN[0001] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument INFO[0000] running as rootless DEBU[0000] using conmon: "/usr/libexec/podman/conmon" DEBU[0000] Initializing boltdb state at /sysroot/home/nazanin/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /sysroot/home/nazanin/.local/share/containers/storage DEBU[0000] Using run root /run/user/1000 DEBU[0000] Using static dir /sysroot/home/nazanin/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp DEBU[0000] Using volume path /sysroot/home/nazanin/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] using runtime "/usr/lib/cri-o-runc/sbin/runc" WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Initialized SHM lock manager at path /libpod_rootless_lock_1000 DEBU[0000] Podman detected system restart - performing state refresh DEBU[0000] overlay: mount_data=lowerdir=/sysroot/home/nazanin/.local/share/containers/storage/overlay/l/NAHCVSUJ4H5CUUW4JQ2GXKGY6O:/sysroot/home/nazanin/.local/share/containers/storage/overlay/l/GZ6DJVVYELMA62HPQ56X6J2KNK,upperdir=/sysroot/home/nazanin/.local/share/containers/storage/overlay/72bdf0b09a81e63900cf1abe6564c0a58a44f19fe91cd24054dc4576556962f3/diff,workdir=/sysroot/home/nazanin/.local/share/containers/storage/overlay/72bdf0b09a81e63900cf1abe6564c0a58a44f19fe91cd24054dc4576556962f3/work DEBU[0001] mounted container "81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f" at "/sysroot/home/nazanin/.local/share/containers/storage/overlay/72bdf0b09a81e63900cf1abe6564c0a58a44f19fe91cd24054dc4576556962f3/merged" DEBU[0001] Created root filesystem for container 81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f at /sysroot/home/nazanin/.local/share/containers/storage/overlay/72bdf0b09a81e63900cf1abe6564c0a58a44f19fe91cd24054dc4576556962f3/merged DEBU[0001] /etc/system-fips does not exist on host, not mounting FIPS mode secret DEBU[0001] set root propagation to "rslave" DEBU[0001] Created OCI spec for container 81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f at /sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/config.json DEBU[0001] /usr/libexec/podman/conmon messages will be logged to syslog DEBU[0001] running conmon: /usr/libexec/podman/conmon args="[--api-version 1 -c 81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f -u 81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f -r /usr/lib/cri-o-runc/sbin/runc -b /sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata -p /run/user/1000/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/pidfile -l k8s-file:/sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /sysroot/home/nazanin/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f]" WARN[0001] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpu: mkdir /sys/fs/cgroup/cpu/libpod_parent: permission denied [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0001] Received: -1
DEBU[0001] Cleaning up container 81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f
DEBU[0001] Network is already cleaned up, skipping…
DEBU[0001] unmounted container “81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f”
ERRO[0001] unable to start container “fedora-toolbox-33”: time=“2021-04-20T18:48:37+04:30” level=warning msg=“exit status 1”
time=“2021-04-20T18:48:37+04:30” level=warning msg=“no such directory for freezer.state”
time=“2021-04-20T18:48:37+04:30” level=warning msg=“no such directory for freezer.state”
time=“2021-04-20T18:48:37+04:30” level=error msg=“container_linux.go:346: starting container process caused “process_linux.go:449: container init caused \“rootfs_linux.go:58: mounting \\\”/dev\\\” to rootfs \\\”/sysroot/home/nazanin/.local/share/containers/storage/overlay/72bdf0b09a81e63900cf1abe6564c0a58a44f19fe91cd24054dc4576556962f3/merged\\\" at \\\"/sysroot/home/nazanin/.local/share/containers/storage/overlay/72bdf0b09a81e63900cf1abe6564c0a58a44f19fe91cd24054dc4576556962f3/merged/dev\\\" caused \\\“operation not permitted\\\”\""\n"
container_linux.go:346: starting container process caused “process_linux.go:449: container init caused “rootfs_linux.go:58: mounting \”/dev\” to rootfs \"/sysroot/home/nazanin/.local/share/containers/storage/overlay/72bdf0b09a81e63900cf1abe6564c0a58a44f19fe91cd24054dc4576556962f3/merged\" at \"/sysroot/home/nazanin/.local/share/containers/storage/overlay/72bdf0b09a81e63900cf1abe6564c0a58a44f19fe91cd24054dc4576556962f3/merged/dev\" caused \“operation not permitted\”"": OCI runtime error

#8

Can you please run the following commands:

podman stop fedora-toolbox-33
sudo rm /sysroot/home/nazanin/.local/share/containers/storage/overlay-containers/81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f/userdata/ctl

Now, try to enter the Toolbox:

toolbox enter --release 33

Does it work now?

1 Like
#9

sadly it didn’t work

when I run the first command I get this :

Error: can only stop created, running, or stopped containers. 81f9cc22be4270c89d2bc92e038682f3d8691da0a3af8d3cd029a1aa68e58f0f is in state configured: container state improper

#10

OK, the first error is expected - the command is just there to make sure that the container is stopped before executing the second command.

Sadly i’m running out of ideas … if you don’t have any special reasons for keeping this toolbox, i would go the route of removing it and simply creating it from scratch.

toolbox rm -f -a

will remove all currently configured toolboxes (which is normally only one).

1 Like
#11

I was just about to ask if I can remove it :smiley:

I should have just done this from the beginning
hopefully I won’t run into any problem while reinstalling my stuff :smiley:

Thank you for all of your help Egarth :pray: :pray: :rose: :rose:

#12

You’re welcome. We both learned from this one :slight_smile:

1 Like
#13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.