1# ICU version upgrade 2 3Upgrade the ICU on Android to the new upstream version. The new upstream versions can be found at 4https://github.com/unicode-org/icu/releases. 5 6The below contains the steps and commands in order to upgrade the ICU version in the AOSP. 7 8# Prerequisites 91. Install the prerequisite tools 10 * See http://site.icu-project.org/repository 11 * [git-lts](https://git-lfs.github.com/) has to be installed to pull binaries correctly from the github. 122. Generate a github read-access token and setup a local ~/.m2/setting.xml 13 * See http://cldr.unicode.org/development/maven. This is required to download 14 the prebuilt ICU to break the circular dependency between CLDR and icu. 153. Check out aosp/main 16 * http://go/repo-init/aosp-main-with-phones 17 18# Steps 191. Create a new branch in AOSP 20 * The branch is a staging branch to **rebase** Android-specific patches onto the new upstream versions. 21 * See detailed instruction at http://g3doc/company/teams/android-build-team/playbook/create_branch#creating-new-git-branches. 22 * **Choose a branch name** 23 * Create aosp/icu{version} branch, e.g. `aosp/icu67` in the `external/icu` project 24 forked from `aosp/main` 25 * For the external/cldr project, we don't need a new branch. 26 We use the **`aosp/upstream-release-cldr** as the mirror of the upstream release branch. 27 We don’t modify this branch with Android patches, but merge this branch 28 into aosp/main where the Android patches are located. 292. Configure the versions and temp directory 30 31 2a. Customize the following environment variables. 32 The following example targets the ICU 71.1 and CLDR 41.0 version. 33 ```shell 34 export ICU_VERSION=71 35 export ICU_MINOR_VERSION=1 36 export CLDR_VERSION=41 37 export ICU_UPGRADE_BUG=1234567890 # buganizer bug 38 # Initially empty directory to store upstream source 39 export UPSTREAM_CLDR_GIT=/media/user/disk/icu-git/cldr 40 export UPSTREAM_ICU_GIT=/media/user/disk/icu-git/icu 41 ``` 42 43 2b. Build the clean source of AOSP 44 ```shell 45 source build/envsetup.sh 46 lunch sdk_phone_x86_64-userdebug 47 m 48 ``` 493. Copy the CLDR sources into the `upstream-release-cldr` branch 50 51 3a. Copy sources 52 ```shell 53 export CLDR_UPSTREAM_BRANCH=release-${CLDR_VERSION} 54 55 cd ${ANDROID_BUILD_TOP}/external/cldr 56 git fetch aosp upstream-release-cldr 57 git branch ${CLDR_UPSTREAM_BRANCH} --track aosp/upstream-release-cldr 58 git checkout ${CLDR_UPSTREAM_BRANCH} 59 60 git clone https://github.com/unicode-org/cldr.git ${UPSTREAM_CLDR_GIT} 61 62 git --git-dir=${UPSTREAM_CLDR_GIT}/.git --work-tree=${UPSTREAM_CLDR_GIT} fetch 63 git --git-dir=${UPSTREAM_CLDR_GIT}/.git --work-tree=${UPSTREAM_CLDR_GIT} checkout ${CLDR_UPSTREAM_BRANCH} 64 rm -rf * 65 cp -r ${UPSTREAM_CLDR_GIT}/* . 66 git clean -dfX # Remove ignored files 67 git add -A 68 git commit -F- <<EOF 69 Copy upstream ${CLDR_UPSTREAM_BRANCH} 70 71 Bug: ${ICU_UPGRADE_BUG} 72 Test: n/a 73 EOF 74 # Upload this CL to upstream-release-cldr branch 75 repo upload --cbr . 76 ``` 77 78 3b. Merge the upstream sources with patches in `aosp/main` 79 ```shell 80 export CLDR_BRANCH=cldr${CLDR_VERSION}-main 81 git branch ${CLDR_BRANCH} --track aosp/main 82 git checkout ${CLDR_BRANCH} 83 git merge ${CLDR_UPSTREAM_BRANCH} -m " 84 Merge CLDR ${CLDR_VERSION} in upstream-release-cldr into aosp/main 85 86 Bug: ${ICU_UPGRADE_BUG} 87 Test: external/icu/tools/updatecldrdata.py 88 " 89 ``` 90 91 3c. Resolve any merge conflicts with the Android-specific patches. 92 Continue creating the merge commit 93 ```shell 94 git merge --continue 95 ``` 96 97 3d. Upload the CL to main branch 98 ```shell 99 repo upload --cbr . 100 ``` 101 1024. Copy ICU upstream sources into external/icu 103```shell 104cd ${ANDROID_BUILD_TOP}/external/icu 105export ICU_BRANCH=icu${ICU_VERSION} 106export UPSTREAM_RELEASE_TAG=release-${ICU_VERSION}-${ICU_MINOR_VERSION} 107git fetch aosp ${ICU_BRANCH} 108git branch ${ICU_BRANCH} --track aosp/${ICU_BRANCH} 109git checkout ${ICU_BRANCH} 110 111# Clone the upstream CLDR repo locally to ${UPSTREAM_ICU_GIT} 112test -d ${UPSTREAM_ICU_GIT} || git clone https://github.com/unicode-org/icu.git ${UPSTREAM_ICU_GIT} 113 114git --git-dir=${UPSTREAM_ICU_GIT}/.git --work-tree=${UPSTREAM_ICU_GIT} fetch 115git --git-dir=${UPSTREAM_ICU_GIT}/.git --work-tree=${UPSTREAM_ICU_GIT} checkout ${UPSTREAM_RELEASE_TAG} 116find icu4j/ -type f,d ! -regex ".*/\(Android.mk\|Android.bp\|adjust_icudt_path.mk\|liblayout-jarjar-rules.txt\|.gitignore\|AndroidTest.xml\)" -delete 117find icu4c/ -type f,d ! -regex ".*/\(Android.mk\|Android.bp\|.gitignore\|AndroidTest.xml\)" -delete 118cp -r ${UPSTREAM_ICU_GIT}/icu4j . 119cp -r ${UPSTREAM_ICU_GIT}/icu4c . 120git checkout HEAD -- icu4c/.gitignore icu4j/.gitignore # Android has extra .gitignores. Use our version. 121rm -r tools/cldr 122cp -r ${UPSTREAM_ICU_GIT}/tools/cldr tools/cldr 123 124git add -A 125git commit -F- <<EOF 126Copy ICU ${UPSTREAM_RELEASE_TAG} into aosp/${ICU_BRANCH} 127 128Copy the files with the following commands: 129find icu4j/ -type f,d ! -regex ".*/\(Android.mk\|Android.bp\|adjust_icudt_path.mk\|liblayout-jarjar-rules.txt\|.gitignore\|AndroidTest.xml\)" -delete 130find icu4c/ -type f,d ! -regex ".*/\(Android.mk\|Android.bp\|.gitignore\|AndroidTest.xml\)" -delete 131cp -r \${UPSTREAM_ICU_GIT}/icu4j . 132cp -r \${UPSTREAM_ICU_GIT}/icu4c . 133git checkout HEAD -- icu4c/.gitignore icu4j/.gitignore 134rm -r tools/cldr 135cp -r \${UPSTREAM_ICU_GIT}/tools/cldr tools/cldr 136EOF 137``` 138 1395. Apply Android-specific patches into `external/icu` 140 141 5a. Cherry-pick the patches from the last staging branch. For example using the following query for ICU 71 patches 142 * https://r.android.com/q/%2522Android+patch%2522+branch:icu71+status:merged 143 * The cherry-pick command is 144 ```shell 145 git cherry-pick <first_patch_in_the_chain>~1..<last_patch_in_the_chain> 146 ``` 147 5b. Cherry-pick the patches since the ICU upgrade 148 * Find the patches with this query. 149 https://r.android.com/q/%2522Android+patch%2522+project:platform/external/icu+status:merged+-owner:automerger+-owner:android-build-coastguard-worker%2540google.com+branch:main 150 1516. Regenerate and commit the artifacts 152 153 6a. Update icu source data files 154 ```shell 155 croot external/icu 156 tools/updatecldrdata.py 157 158 git add -A 159 git commit -F- <<EOF 160 Regenerated source data files with Android CLDR patches 161 162 Source data files updated using: 163 tools/updatecldrdata.py 164 165 Test: n/a 166 EOF 167 ``` 168 169 6b. Update icu binary data files 170 ```shell 171 tools/updateicudata.py 172 git add -A 173 git commit -F- <<EOF 174 Regenerated binary data files with Android CLDR patches 175 176 Binary data files updated using: 177 tools/updateicudata.py 178 179 Test: n/a 180 EOF 181 ``` 182 183 6c. Pin the public API surface temporarily 184 * We will later expose the new APIs after submitting the version upgrade CLs. 185 ```shell 186 ./tools/srcgen/generate_allowlisted_public_api.sh 187 git add -A 188 git commit -F- <<EOF 189 Pin the current API list 190 191 Test: ./tools/srcgen/generate_allowlisted_public_api.sh 192 EOF 193 ``` 194 195 6d. Regenerate android_icu4j/ 196 * Commands 197 ```shell 198 tools/srcgen/generate_android_icu4j.sh 199 200 git add -A 201 git commit -F- <<EOF 202 Regenerate android_icu4j/ from icu4j/ 203 204 android_icu4j files updated using: 205 tools/srcgen/generate_android_icu4j.sh 206 207 Test: n/a 208 EOF 209 ``` 210 * If any java patches are not applied perfectly, remove the .orig and 211 .rej files with the following command 212 ```shell 213 find android_icu4j/ -name *.orig -delete 214 # Please manually update and resolve conflict of the java doc in android_icu4j/ 215 git commit -amend 216 # Regenerate the patch files 217 ./tools/srcgen/javadoc_patches/create_patches.sh 218 git add -A && git commit -F- <<EOF 219 Regenerate java doc patches 220 221 Test: n/a 222 EOF 223 ``` 224 225 6e. Re-genereate libandroidicu/ sources 226 ```shell 227 ./tools/icu4c_srcgen/generate_libandroidicu.py 228 git add -A && git commit -F- <<EOF 229 Regenerate libandroidicu 230 231 The command: 232 ./tools/icu4c_srcgen/generate_libandroidicu.py 233 234 Test: n/a 235 EOF 236 ``` 237 238 6f. Regenerate libicu/ sources 239 * Commands 240 ```shell 241 ./tools/icu4c_srcgen/generate_ndk.py 242 243 git add -A && git commit -a -F- <<EOF 244 Regenerate libicu.so and ICU4C CTS headers 245 246 The command: 247 ./tools/icu4c_srcgen/generate_ndk.py 248 249 Test: n/a 250 EOF 251 ``` 252 * Review the changes in libicu/ndk_headers/unicode. Check that 253 no ABI change is in the C struct or API signature. 254 ```shell 255 git diff HEAD~1 ./libicu/ndk_headers/ 256 ``` 257 * May need to run this to update .patch files 258 ```shell 259 # Manually update doxygen doc in the .h headers in libicu/ndk_headers 260 ./tools/icu4c_srcgen/doc_patches/create_patches.sh 261 git add -A && git commit -F- <<EOF 262 Regenerate patches in libicu headers 263 EOF 264 ``` 265 266 6g. [Not required for every ICU release] Increment Distro major version 267 * See https://android.googlesource.com/platform/system/timezone/+/main/README.android 268 for details. Usually, it’s needed only when it’s the first ICU upgrade 269 in the Android dessert release. 270 271 6h. Generate time zone files 272 ```shell 273 cd $ANDROID_BUILD_TOP/system/timezone 274 repo start ${ICU_BRANCH} . 275 ./update-tzdata.py 276 git add -A 277 git commit -F- <<EOF 278 Regenerate data files for ICU ${ICU_VERSION} upgrade 279 280 Binary data files updated using: 281 system/timezone/update-tzdata.py 282 283 This updates the ICU version number inside of icu_tzdata.dat but doesn't 284 change the timezone data itself. 285 286 Bug: ${ICU_UPGRADE_BUG} 287 Test: n/a 288 EOF 289 290 # If only the build time in icu4c/source/data/misc/zoneinfo64.txt, this step isn't needed. 291 cd $ANDROID_BUILD_TOP/external/icu 292 git add -A 293 git commit -F- <<EOF 294 Regenerate tz-related data files 295 296 Data files updated using: 297 system/timezone/update-tzdata.py 298 299 Bug: ${ICU_UPGRADE_BUG} 300 Test: n/a 301 EOF 302 ``` 303 304 6i. Update the version numbers in the METADATA 305 1. Update the external/icu/README.version 306 2. Update version and upgrade date in external/cldr/METADATA 307 3. `git commit` the file change 308 309 6j. Regenerate frameworks/base/libs/androidfw/LocaleDataTables.cpp 310 ```shell 311 croot frameworks/base 312 ./tools/localedata/extract_icu_data.py $ANDROID_BUILD_TOP > libs/androidfw/LocaleDataTables.cpp 313 git commit -a -F- <<EOF 314 Regenerate LocaleDataTables.cpp due to ICU ${ICU_VERSION} upgrade 315 316 The command: 317 ./tools/localedata/extract_icu_data.py \$ANDROID_BUILD_TOP > libs/androidfw/LocaleDataTables.cpp 318 319 Bug: ${ICU_UPGRADE_BUG} 320 Test: atest FrameworksCoreTests:android.text.format 321 EOF 322 ``` 3237. Build and run test 324 325 7a. Build by `m droid cts` 326 327 7b. Run the device tests by `atest CtsIcu4cTestCases CtsIcuTestCases CtsLibcoreTestCases CtsLibcoreOjTestCases CtsBionicTestCases CtsTextTestCases minikin_tests -- --abi x86_64 # the primary ABI` 328 329 7c. Run the host-side test by 330 * ICU4J host-side test `ant check` 331 * ICU4C host-side test `make CINTLTST_OPTS=-w INTLTEST_OPTS=-w check` 332 (Currently, it has some failing tests. No of failures?) 333 334 7d. If libcore/ tests are changed, verify the change 335 * To verify the test change in ART MTS, run `m mts && mts-tradefed run mts-art` on Android S. 336 * To verify the test change on LUCI bot, run `art/tools/run-libcore-tests.sh --mode=host`, because LUCI uses older ICU versions. 3378. Upload the CLs to gerrit for code reviews from `aosp/icu${ICU_VERSION}` in `external/icu` and `aosp/upstream-release-cldr` in `external/cldr` 338```shell 339repo upload --cbr -o uploadvalidator~skip --no-verify . 340``` 3419. Merge `aosp/icu*` branch to aosp/main 342```shell 343cd $ANDROID_BUILD_TOP/external/icu 344repo start icu${ICU_VERSION}-main . 345git merge --no-ff icu${ICU_VERSION} -m " 346Merge branch aosp/icu${ICU_VERSION} into aosp/main 347 348Bug: ${ICU_UPGRADE_BUG} 349Test: atest CtsIcu4cTestCases CtsIcuTestCases CtsLibcoreTestCases CtsLibcoreOjTestCases CtsBionicTestCases CtsTextTestCases minikin_tests 350" 351``` 35210. Upload and submit changes from external/icu, external/cldr, libcore, frameworks/base, system/timezone 353 354 10a. `repo upload --cbr -o uploadvalidator~skip --no-verify .` 355 356 10b. Code review the diff in `android_icu4j/src/main/tests/android/icu/extratest/expected_transliteration_id_list.txt` 357 * If a transliteration id is removed from the list, it may introduce 358 app compatibility issues if the app depends on them. However, app 359 has been warned to check the availability before invoking them. 360 https://developer.android.com/reference/android/icu/text/Transliterator#getAvailableIDs() 36111. After submitting all the CLs to aosp/main, expose the new stable ICU4J APIs to Android SDK 362```shell 363rm tools/srcgen/allowlisted-public-api.txt 364./tools/generate_android_icu4j.sh 365# Modify Icu4jTransform.java to allowlist more classes if needed. Check the error message for the details 366m update-api droid 367 368git commit -a -F- <<EOF 369Expose the new stable APIs from ICU4J ${ICU_VERSION} 370 371According to the upstream API coverage report external/icu/icu4j/coverage-exclusion.txt, 372the methods should have API coverage running in the existing CtsIcuTestCases. 373 374Bug: ${ICU_UPGRADE_BUG} 375Test: atest CtsIcuTestCases 376EOF 377``` 37812. Send email android-libcore@ and icu-team@ to announce this. 379 1. Some Android teams may have dependency on the new ICU version for other features. 380 2. Email template 381 ```text 382 Hi, Libcore and ICU team, 383 384 ICU <version> just landed Android AOSP. 385 https://android.googlesource.com/platform/external/icu/+/main/README.version 386 as well as Android S (or Android 12). 387 https://googleplex-android.googlesource.com/platform/external/icu/+/sc-dev/README.version 388 389 Note: 390 - Contains bug fixes / build changes with a small set of API methods added. 391 - Unicode stays at version 14, and no new version has been published yet. 392 ``` 393 394 395## Historic version of this doc 396http://g3doc/third_party/icu/g3doc/update/android