Endless Mini new installation - missing ARM image

#1

I recently obtained an Endless Mini computer. It’s an ARM-based ball-like looking mini computer which hasn’t been updated for 3 years (see the attached eos-diagnostic).

Unfortunately I didn’t manage to update any part of the system because the repos are not available and if I use newer repos (manually added), it’s full of errors (version issues, dependencies issues - mostly due to changed names, GPG issues I couldn’t overcome as old keys are gone and/or expired, etc.) and nothing gets installed.

I discovered, that the Endless OS version installed uses one of the very first releases of flatpak which didn’t offer any modern capabilities useful for emergency cases like this one. And because everything is Flatpak, I can’t even update the underlying system using standard Debian tools and then switch back to using Flatpak.

After about 16 hours of trying (incl. special upgrade scripts Endless created - see https://github.com/endlessm/eos-customer-support ) I gave up and decided to install Endless OS from scratch. To my greatest surprise, there are only x86(-64) images available, but no ARM images as it seems Endless Mini is the only non-x86 computer Endless ever made. I didn’t find any notice about discontinued support for Endless Mini though in any of the tens of announcements of releases of Endless OS since 2016. So what is this for a fishy game?

So I’m here to ask for help. At best just download link with an ARM image for Endless Mini and short instructions how to install it (surprisingly I couldn’t find any disassembly info nor any technical boot information about Endless Mini).

eos-diagnostic-191101_162253_UTC 0100.txt (564.9 KB)

#2

Sorry for the trouble here!

As an embedded device, reinstalling the Endless Mini image is a somewhat involved process, its not like the PC software reinstallation.

It’ll be much easier to investigate and overcome the upgrade difficulty and then do an online update.

Flatpak will be updated as part of the OS so we can skip over that part of the challenge. We just need to make the underlying ostree update. The logs show that it’s failing on GPG as you mentioned.

Could you please post the output of ls -l /usr/share/ostree/trusted.gpg.d

#3

Another thing to check here…

In the old OS versions like this one, in order to update the base OS you have to go to Settings->Details (or wait several days for the automatic upgrade to kick in). Have you tried doing that?

What about if you run ostree admin upgrade as root, what is the output there?

#4

As an embedded device, reinstalling the Endless Mini image is a somewhat involved process, its not like the PC software reinstallation.

I’m an embedded Linux engineer, so I’m not afraid of it. But let’s try first the ostree magic (I’m though still convinced, that there should be images for ARM available as well because Endless Mini is an officially supported device and corruption or other stuff just happens…).

Could you please post the output of ls -l /usr/share/ostree/trusted.gpg.d


user@endless:~$ ls -1 /usr/share/ostree/trusted.gpg.d/
eos-flatpak-keyring.gpg
eos-ostree-keyring.gpg
README-gpg
user@endless:~$



user@endless:~$ sudo ostree -v admin upgrade
OT: Deployment c95393cf224054f171b5c633203c7593f9ed00c9e11dcde64c3b04e0a0638c19.0 unlocked=0

OT: resuming legacy transaction
OT: processing delta superblock for abbc01903f9b6f2ad4a9dd8ca5f0b2f32b7251c7d43e3cdeb17223de1d0bc5a5-dcfb6c74046638750f370d12a327152055f3510e255b70a11a95cf5fc112104d
OT: fetch static delta part 7dc1b328257f0e90ff8f05ca05d9cdb1d8f92cae6865e51e70590aaa1be7cd07 complete
Writing objects: 1                                                              OT: execute static delta part 7dc1b328257f0e90ff8f05ca05d9cdb1d8f92cae6865e51e70590aaa1be7cd07 complete

error: opcode set-read-source: No such file object a2c3c27e05c520c8731fbf9f8a8a5fc2a357673d3806f6ecbcecedc8f2f36ee5
user@endless:~$
#5

I wonder if ostree fsck can detect any problems here.

Please also try running this script as root, it should restore any missing objects: https://raw.githubusercontent.com/endlessm/eos-meta/master/eos-tech-support/eos-fix-ostree-repo

#6

Thanks Daniel. Here are the results (in this order).


user@endless:~$ sudo ostree -v fsck
[sudo] password for user:

... (hundreds of messages like the following last two)

OT: Found dirtree object d5574bbf47a8f5ce3797cf18ab40d19d0c671d8a4129c51a3627da4fe66e9bed
OT: Found dirmeta object 446a0ef11b7cc167f3b603e585c7eeeeb675faa412d5ec73f62988eb0b6c5488
1/99282 objects
9929/99282 objects
19857/99282 objects
29785/99282 objects
39713/99282 objects
49641/99282 objects
59569/99282 objects
69497/99282 objects
79425/99282 objects
89353/99282 objects
99281/99282 objects
1 partial commits not verified
user@endless:~$


user@endless:~$ sudo ./eos-fix-ostree-repo

[sudo] password for user:
WARNING: Do not start App Center while this is running
Killing processes with /sysroot/ostree/repo open with signal 15
Killing processes with /sysroot/ostree/repo open with signal 9
Fixing refs pointing to missing commits in /sysroot/ostree/repo
Marking commits with missing objects as partial in /sysroot/ostree/repo
Commit abbc01903f9b6f2ad4a9dd8ca5f0b2f32b7251c7d43e3cdeb17223de1d0bc5a5 already marked as partial
/usr/lib/python3/dist-packages/gi/overrides/GLib.py:253: Warning: g_variant_unref: assertion 'value->ref_count > 0' failed
  self.unref()
Pulling abbc01903f9b6f2ad4a9dd8ca5f0b2f32b7251c7d43e3cdeb17223de1d0bc5a5 commit from eos for os/eos/ec100/eos3
Traceback (most recent call last):
  File "./eos-fix-ostree-repo", line 415, in <module>
    main()
  File "./eos-fix-ostree-repo", line 409, in main
    pull_partial_refs(repo)
  File "./eos-fix-ostree-repo", line 324, in pull_partial_refs
    pull_commit(repo, remote, checksum, full=True)
  File "./eos-fix-ostree-repo", line 220, in pull_commit
    repo.pull_with_options(remote, opts, progress)
GLib.Error: g-io-error-quark: GPG signatures found, but none are in trusted keyring (0)
user@endless:~$ echo $?
1
user@endless:~$
#7

Hmm, sorry about that. Looks like an ostree object is missing and that usual script wasn’t able to recover it as the old commit you are running was signed with an expired key.

Can you try this:
ostree pull --disable-static-deltas eos os/eos/ec100/eos3

If that works then run again:
ostree admin upgrade

#8

Hm, didn’t know about such ostree pull magic. Either way here are the current results.


user@endless:~$ sudo ostree pull --disable-static-deltas eos os/eos/ec100/eos3
[sudo] password for user:

GPG: Verification enabled, found 2 signatures:

  Signature made Út 14. leden 2020, 23:07:58 CET using RSA key ID 9E08D8DABA02FC46
  BAD signature from "EOS OSTree Signing Key 1 <maintainers@endlessm.com>"

  Signature made Út 14. leden 2020, 23:08:00 CET using RSA key ID FCF17B17F1F8E157
  Good signature from "EOS Flatpak Signing Key 1 <maintainers@endlessm.com>"

810 metadata, 6792 content objects fetched; 188004 KiB transferred in 133 seconds
user@endless:~$ echo $?
0
user@endless:~$ sudo ostree admin upgrade

4 metadata, 0 content objects fetched; 3734 KiB transferred in 2 seconds        
Copying /etc changes: 19 modified, 0 removed, 28 added
Transaction complete; bootconfig swap: yes deployment count change: 1
Freed objects: 31,2 MB
user@endless:~$ echo $?
0
user@endless:~$
#9

That looks successful! Reboot and you will hopefully be on version 3.3.20, which is a checkpoint release along the upgrade path.

Then you can upgrade again - I think through the Settings->Details UI - and you should get to 3.7.

#10

Hi Daniel, so I did that and eos-diagnostic shows:

#11

I think your message got cut short, I don’t see the diagnostic info.

#12

It seems so - I’ve always replied from email and it worked, but it didn’t this time (the sent email looks correct, but here nothing shows except for the first sentence). Here it comes:

eos-diagnostic-200201_103240_UTC 0100.txt (327.3 KB)

If it’s correct, then thank you for helping me out. Now I’ll try installing some software using flatpak and see if that goes well.

Why there are no ARM images available? Will it change any time soon? I just noticed you’ve added support for Raspberry Pi 4 - so I still don’t understand why not for your official hardware Endless Mini…

#13

Looks correct, it’s updated to 3.7.6 and the flatpak remote configuration looks sane at a glance.

There aren’t any plans to publish the image, because it’s unfortunately painful to install due to SoC limitations. You have to either do some weird initramfs stuff, or alternatively erase specific important eMMC sectors which makes the device look like a brick, and then physically open the case (which is surprisingly difficult) and then do a weird dance with the internal SD card. As an engineer I think you would have managed (and I would have guided you through this if we had not found this other solution), but it’s not something for the general public, and would have taken more time than the steps taken here. Plus I was curious in understanding the causes behind the automatic update failing which you have helped me to do :slight_smile:

#14

Well, the install procedure you roughly sketched doesn’t look any more complex than with other SBCs etc. But anyway, my problem got solved and I can only recommend you as a particularly helpful and kind person :wink:.

#15

Ok, that was a bit premature judgment :frowning: . The graphics doesn’t start any more - see the following in the attached log.

...
endless gdm-Xorg-:5[1107]: (EE) AIGLX error: dlopen of /usr/lib/arm-linux-gnueabihf/dri/armsoc_dri.so failed (/usr/lib/arm-linux-gnueabihf/dri/armsoc_dri.so: cannot open shared object file: No such file or directory)
...
endless gdm-Xorg-:5[1107]: Fatal server error:
endless gdm-Xorg-:5[1107]: (EE) Caught signal 11 (Segmentation fault). Server aborting
...

eos-diagnostic-200213_220106_UTC 0100.txt (300.8 KB)

#16

The AIGLX message is expected, but the segfault is not. I checked 3.7.7 on an Endless Mini and it works fine, so that does indeed point at something rather strange happening on your device.

I wonder if ostree fsck could catch it…

Anyway, if the device is under warranty you could reach out to our customer support team to see if you can return it for reflash or replacement.

Alternatively send me a private message on the forum and I’ll send you some reflash instructions. This will void any warranty that is still standing.