BUG? Cannot read /storage: permission denied - Android 11

[nathaneltitane]

Problem description

Previously, using 'termux-setup-storage' to enable storage permissions and accessing /storage for reading or listing was permitted.
I use that method to provide access to my external microSD card and link it to whaterver Termux/CHROOT location i desire.
Now that I have upgraded my device to Android 11, after enabling storage permissions (by script or manually, an error results after attempting to read /storage:

example error output:

ˋˋˋ$ ls /storage
ls: cannot open directory '/storage': Permission denied
$ˋˋˋ

Steps to reproduce

open up a fresh/clean instance of termux
run ˋtermux-setup-storageˋ manually or programatically
allow permissions
attempt reading /storage directory contents manually or programatically (i.e.: ˋls storageˋ
error

Expected behavior

If storage permissions are allowed for tha app to access and manipulate storage, no error hould be triggered when attempting to access previously unauthorized or restricted storage directories (as was the behavior prior to Android 11 in my case)

Additional information

• Termux application version: latest
• Android OS version: Android 11
• Device model: Samsung Galaxy Tab S7+ SM-T970

[ghost]

This is not a bug. System directories are being locked down as really shouldn't be viewable by regular apps. Least privilege principle, which is being enforced on modern Android.

[ghost]

Use ls /sdcard or ls /storage/emulated/0, i.e. direct path to readable directory.

If /sdcard somehow became unreadable (permission denial), then revoke and grant again storage permission.

[nathaneltitane]

how does one detect uids of multiple sdcard-like devices or other forms of external storage then?

EDIT:

@xeffyr

example, if i have an actual micro sdcard connected AND an external SSD drive for instance...

[ghost]

External sdcard mount path can be detected as path component of ~/storage/external-1 symlink target. However there no ways for USB storage. Either downgrade to older Android or use root.

In future Termux versions, with updated SDK level to 30+, support of external storage will be dropped as it literally useless even with write access to application private directory (files placed in it cannot be moved elsewhere or even accessed with external file manager app).

[nathaneltitane]

thanks for clarification

[RalfWerner]

Termux versions, with updated SDK level to 30+, support of external storage will be dropped

@xeffyr Have you completed your MANAGE_EXTERNAL_STOREAGE test?
I can currently only check Android 11 (30+) on emulators and I can confirm the missing termux access from 29+ - also with your last commit, but not the missing fileApp access (SAF), including the most internal termux data!

In practice (support will be dropped) this means: all relevant project data must be copied with the fileApp to a location under path: t (ln -s ~ /../../ t) and since this data is disposed of with termux beforehand be saved with the fileApp extern.

Everything (relevant) is then stored at least twice on the device. Most of my real devices are equipped with 16Gb (of which a maximum of 5Gb free) and "sufficient" external storage in which media/downloads (project data ~20Gb) are already managed.

[ghost]

Have you completed your MANAGE_EXTERNAL_STOREAGE test?

If you read my messages on Gitter about it, then you will know that yes and it works. You need to grant a special permission to get it working: Settings -> Applications -> Special access -> Access all files permission.

You can store files in Termux directories and in /storage/emulated/0. On external storage, you can continue to use app private directory as writable location but you will not be able to move files anywhere else (Android/data/com.termux can be accessed only by Termux or ADB).

[ghost]

@nathaneltitane Perhaps a way to get UUIDs is still present. Access to /proc/mounts is unrestricted, so you can list mountpoints under /storage and other directories.

[agnostic-apollo]

The files created in external storage by users for termux-only usage like /storage/emulated/0/ directory would also be deleted if termux is uninstalled when targeting sdk 29(android 10) like even the termux tar backup. We need to add the hasFragileUserData flag to the AndroidManifest.xml to show the user a prompt of whether he wants to delete termux data on uninstall, I don't currently see it. A guide on how to add the flag is here. Note that the data will still be removed if its uninstalled from playstore instead of settings app regardless of the flag value, since playstore uninstalls in the background without any prompts, at least that what the link says, maybe play store app update has fixed that now.

[ghost]

@agnostic-apollo Is this same when MANAGE_EXTERNAL_STORAGE is in effect? Target API 30, of course.

As I know, scoped storage works only if application doesn't has permission for all files access.

<uses-permission
        android:name="android.permission.MANAGE_EXTERNAL_STORAGE"
        tools:ignore="ScopedStorage" />
<uses-permission
        android:name="android.permission.WRITE_EXTERNAL_STORAGE"
        android:maxSdkVersion="29" />

and android:requestLegacyExternalStorage="true"

[ghost]

Can't reproduce behaviour (all app-specific files deleted when uninstalling app) with manifest flags as in #1926 (comment)

[agnostic-apollo]

So I was a mistaken, that flag shouldn't affect any files under /storage/emulated/0/ except app private directories in /storage/emulated/0/Android/{data,media,obb}/. The documentation for hasFragileUserData should mention this. And it engages regardless of MANAGE_EXTERNAL_STORAGE or scoped storage or target sdk on target sdk android >=10, don't see any checks in PackageParser, ApplicationInfo.hasFragileUserData() or while creating the UninstallAlertDialogFragment.

Edit:
It seems that (one of) the check happens during build time, android studio (AAPT) does not let you compile the apk and shows the Unknown Attribute warning in editor and throws AAPT: error: attribute android:hasFragileUserData not found. exception during build time if the targetsdk is less than the api level in which the property was added. The hasFragileUserData is under the android 10 section in public.xml (and also exists in attrs_manifest.xml), not sure how AAPT actually parses the section and if it even has any significance. Not even sure what happens if you manually patch the apk AndroidManifest.xml file after compilation.

Edit 2: Just incrementing compileSdk version instead of targetSdk version has fixed the warning.

Edit 3: I have created a hasFragileUserData gist with more info.

As I know, scoped storage works only if application doesn't has permission for all files access.

True, although possibly other android 11 restrictions like not being able to access private directories of other apps Android/data/ seems to engage on android 11 regardless of scoped storage status as mentioned here but can be bypassed on target sdk 10 for now.

Regardless of target SDK, Storage Access Framework on Android 11 cannot be used to gain access to Android/data and Android/obb directories.

We haven't opted out of scoped storage either, since requestLegacyExternalStorage is not set to true in the AndroidManifest.xml in the android-10 branch, which would only last till ~nov this year, if we manage to target sdk 30 for playstore releases, and adb commands can "probably" be used to opt out for now without the flag anyways as mentioned here.

adb shell cmd appops set com.termux android:legacy_storage allow && \
adb shell am force-stop com.termux

Anyways, with the hasFragileUserData flag set, if you ask the dialog to preserve the data, then the internal app private directory /data/data/ and external app private directories Android/{data,media,obb}/ would be retained. There is an xda article on this as well here. This may or may not be something we want, but could be useful if users want to write to Android/data, sppecially when targeting sdk 30 and could be useful for switching from playstore to F-Droid or custom builds too, specially for devs, so I guess leave the flag enabled. Are you sure all private data is getting deleted as well for you?

The flag would have been added because since app devs are being pushed to write data to their own private app data directories on external storage instead of random directories, so android provided a way for data to be preserved on uninstall.

Internal details

If you ask the dialog to preserve the data, then DELETE_KEEP_DATA is passed to PackageInstallerService.uninstall(). Ultimately, this reaches removePackageDataLIF() which calls the native destroyAppData() to do the actual deletion, which is only deleting app user specific paths.

Apologies for the misinformation.

[nathaneltitane]

everything you've all just mentioned is very nice but, does it change anything to what @xeffyr gave to me as the final answer?

also, one must also consider the use case where no adb console can be accessed, as the Android device is used as the main PC (I.e.: my use case)

[RalfWerner]

@nathaneltitane No as long as you use the Termux artifact with targetSDK=28 :)

[nathaneltitane]

@nathaneltitane No as long as you use the Termux artifact with targetSDK=28 :)

I don't delve into versioning of termux - I simply use the version available on PlayStore like most people would.

Guess I'll rewrite the detection script to find the mounts under /storage using /proc/mounts like @xeffyr suggested.

I just hope that it'll stay accessible throughout

Thanks

[RalfWerner]

last play store version is v101 (Nov. 2020) the current ones are published in F-Droid and are currently at v107 - in "Notificatons" above, annoying messages can be suppressed.

If you read my messages

Yes read @xeffyr! you have pushed i asked checked if so where (emulator) and with which artifact (branch android-10)?

[nathaneltitane]

So here are the two storage units of intterest:

cat /proc/self/mountinfo | tail -n 2
4473 4469 0:113 / /storage/3211-BCAD rw,nosuid,nodev,noexec,noatime master:57 - fuse /dev/fuse rw,lazytime,user_id=0,group_id=0,allow_other
6587 4362 8:65 / /mnt/media_rw/27E0-7084 rw,nosuid,nodev,noexec,noatime master:60 - sdfat /dev/block/vold/public:8,65 rw,fs=exfat,gid=1077,fmask=0007,dmask=0007,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,utf8,namecase=0,adj_req,symlink=0,bps=512,errors=remount-ro

As mentioned above, i was able to trace back the SDCard 3211-BCAD to /sdcard and or /mnt/sdcard
The external SSD (27E0-7084) seems to mount but the location access rights are restricted... there is no way to access the mount although external file managers are able to read and write to it?
How can it be that termux cannot handle such storage when other applications do?

[ghost]

@nathaneltitane Storage properties are clarified in https://wiki.termux.com/wiki/Internal_and_external_storage. You can have full access only to internal and shared storage.

Only applications using Storage Access Framework APIs are able to write to external storage. Termux command line tools, like bash or coreutils, are not using Android APIs and in general not part of Android OS runtime. They are executed by Linux kernel and are subject for restrictions implied by DAC (chmod/chown) and MAC (SELinux).