xref: /aosp_15_r20/external/google-cloud-java/TROUBLESHOOTING.md (revision 55e87721aa1bc457b326496a7ca40f3ea1a63287) 
1# Troubleshooting
2
3## Logging
4
5Our libraries use the Java logging API via `java.util.logging` package.
6Configuring logging level reveals various facts that help your troubleshooting,
7including:
8
9- The timing of underlying client-server communication
10- Request and response message headers
11- Verbose messages in underlying dependency libraries
12
13While there are various ways to configure the logging,
14to quickly enable verbose logging for the Google Cloud Java libraries, create
15a file `logging.properties` with the following content:
16
17```
18# run java program pointing to this properties file with the java arg
19#   -Djava.util.logging.config.file=path/to/logging.properties
20handlers=java.util.logging.ConsoleHandler
21java.util.logging.SimpleFormatter.format=%1$tF %1$tT,%1$tL %4$-8s %3$-50s - %5$s %6$s%n
22
23# --- ConsoleHandler ---
24java.util.logging.ConsoleHandler.level=ALL
25java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
26.level=INFO
27
28# --- Specify logging level for certain packages ---
29# com.google.api is for HTTP 1.1 layer
30com.google.api.level=ALL
31# io.grpc is for gRPC + Netty layer
32io.grpc.level=FINE
33# com.google.auth is for authentication
34com.google.auth.level=FINE
35
36# Example when we want to specify storge library's level. This works when
37# the target Cloud library uses the logging API.
38com.google.cloud.storage.level=INFO
39```
40
41and run your application with `-Djava.util.logging.config.file=path/to/logging.properties`
42as the "VM argument" (not "Program argument").
43
44If you use IntelliJ, you specify the VM argument in "Run/Debug Configuration":
45
46![Screenshot of IntelliJ configuration](docs/logging_vm_options.png)
47
48If the JVM of your program is running with the configuration correctly, you see
49the FINE-level logging in your console. Example output:
50
51```
522023-04-05 13:03:01,761 FINE     com.google.auth.oauth2.DefaultCredentialsProvider  - Attempting to load credentials from well known file: /usr/local/google/home/suztomo/.config/gcloud/application_default_credentials.json
532023-04-05 13:03:01,847 FINE     io.grpc.ManagedChannelRegistry                     - Unable to find OkHttpChannelProvider
54java.lang.ClassNotFoundException: io.grpc.okhttp.OkHttpChannelProvider
55	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
56	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
57	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
58	at java.base/java.lang.Class.forName0(Native Method)
59	at java.base/java.lang.Class.forName(Class.java:315)
60...
61```
62
63Note that you may see many stacktraces printed there.
64As long as it's "FINE" level, they are not errors.
65
66This is just one way to configure logging level.
67For more details about Java logging API usage, see [Java Logging Overview](
68https://docs.oracle.com/javase/8/docs/technotes/guides/logging/overview.html).
69
70## ALPN is not configured properly
71
72If you see exceptions related to `ALPN is not configured properly`, such as:
73
74```
75Caused by: java.lang.IllegalArgumentException: ALPN is not configured properly. See https://github.com/grpc/grpc-java/blob/master/SECURITY.md#troubleshooting for more information.
76```
77
78Please use the [compatibility checker](https://github.com/googleapis/google-cloud-java/tree/master/google-cloud-util/google-cloud-compat-checker) to see if your environment is compatible with grpc-based clients. The incompatibility can mean that:
79- You are not on a [supported platform](https://github.com/googleapis/google-cloud-java/#supported-platforms).
80- There are classpath conflicts with `netty`.
81- Or, you are seeing any of the conflicts specified in [gRPC Troubleshooting guide](https://github.com/grpc/grpc-java/blob/master/SECURITY.md#troubleshooting).
82
83## ClassNotFoundException, NoSuchMethodError, NoClassDefFoundError
84
85These errors are usually caused by having multiple versions or conflicting versions of the same dependency in the classpath.
86Usually these dependency conflicts occur with `guava` or `protobuf-java`.
87
88There may be multiple sources for classpath conflicts:
89- Multiple versions of the same transitive dependency in the dependency tree
90- Your runtime classpath has different versions of dependencies than what you specified in the build
91
92For example, if you have a direct or a transitive dependency on Guava version 19.0,
93and `google-cloud-java` uses Guava version 30.0,
94then `google-cloud-java` could be using Guava methods that don't exist in Guava 19.0,
95and cause `NoSuchMethodError`.
96
97Similarily, if your classpath has an older version of `protobuf-java`,
98but `google-cloud-java` requires a newer version,
99 then you might see `NoClassDefFoundError` that fails to initialize `google-cloud-java` classes. For example:
100
101```
102java.lang.NoClassDefFoundError: Could not initialize class com.google.pubsub.v1.PubsubMessage$AttributesDefaultEntryHolder
103```
104
105### Validate the conflict
106
107Check the dependency tree to see if you have multiple versions of the same dependencies:
108
109```
110$ mvn dependency:tree
111```
112
113Look for versions of potentially conflicting dependencies like `guava`, `protobuf-java`, etc.
114
115If you experience the error only during runtime, then your runtime environment
116might  be introducing conflicting JARs into your runtime classpath. A typical case
117is that Hadoop, Spark, or other server software that your application runs on
118has conflicting versions `netty`, `guava`, or `protobuf-java` JARs in the classpath.
119
120### Detecting conflicts during build
121
122To detect dependency linkage errors at compile time, add the
123[Linkage Checker Enforcer Rule](https://github.com/GoogleCloudPlatform/cloud-opensource-java/tree/master/enforcer-rules)
124in your pom.xml:
125
126```
127      <plugin>
128        <groupId>org.apache.maven.plugins</groupId>
129        <artifactId>maven-enforcer-plugin</artifactId>
130        <version>3.0.0-M3</version>
131        <dependencies>
132          <dependency>
133            <groupId>com.google.cloud.tools</groupId>
134            <artifactId>linkage-checker-enforcer-rules</artifactId>
135            <version>1.5.7</version>
136          </dependency>
137        </dependencies>
138        <executions>
139          <execution>
140            <id>enforce-linkage-checker</id>
141            <!-- Important! Should run after compile -->
142            <phase>verify</phase>
143            <goals>
144              <goal>enforce</goal>
145            </goals>
146            <configuration>
147              <rules>
148                <LinkageCheckerRule
149                    implementation="com.google.cloud.tools.dependencies.enforcer.LinkageCheckerRule"/>
150              </rules>
151            </configuration>
152          </execution>
153        </executions>
154      </plugin>
155```
156
157There is no way to detect runtime classpath conflicts though. You need to be fully
158aware of which JARs/classes are included in the runtime classpath
159as every server environment is different.
160
161### Resolving the conflict
162
163There are different strategies to resolve conflicts, but you must understand the root cause of the conflicts. For example:
164
165- If you have control over the dependency tree, upgrade
166  offending dependencies (for example, upgrading Guava version). This is the
167  least hackish approach, but it is a lot of work that can require multiple releases
168   of multiple libraries to sync everything up.
169- If you can't modify and push new versions of your dependencies, import
170  `com.google.cloud:libraries-bom:25.1.0` (or a more recent version) and use that to
171  select consistent dependency versions. This is the easiest route.
172  For example, this is how you can depend on consistent versions of Guava and
173  `com.google.cloud:google-cloud-storage` without explicitly setting the version of either one:
174
175```
176  ...
177  <dependencyManagement>
178    <dependencies>
179      <dependency>
180        <groupId>com.google.cloud</groupId>
181        <artifactId>libraries-bom</artifactId>
182        <version>25.1.0</version>
183        <type>pom</type>
184        <scope>import</scope>
185       </dependency>
186     </dependencies>
187  </dependencyManagement>
188  ...
189  <dependencies>
190    <dependency>
191      <groupId>com.google.cloud</groupId>
192      <artifactId>google-cloud-storage</artifactId>
193    </dependency>
194    <dependency>
195      <groupId>com.google.guava</groupId>
196      <artifactId>guava</artifactId>
197    </dependency>
198    ...
199  </dependencies>
200  ...
201```
202
203- If changing dependency versions causes other failures,
204  consider [shading dependencies](https://maven.apache.org/plugins/maven-shade-plugin/)
205  that conflict with `google-cloud-java`.
206
207  For example, to shade `guava` and `protobuf-java`:
208
209```
210  <plugin>
211    <groupId>org.apache.maven.plugins</groupId>
212    <artifactId>maven-shade-plugin</artifactId>
213    <version>...</version>
214    <executions>
215      <execution>
216        <phase>package</phase>
217        <goals>
218          <goal>shade</goal>
219        </goals>
220        <configuration>
221          <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
222          <relocations>
223            <!-- move protobuf to a shaded package -->
224            <relocation>
225              <pattern>com.google.protobuf</pattern>
226              <shadedPattern>myapp.shaded.com.google.protobuf</shadedPattern>
227            </relocation>
228            <!-- move Guava to a shaded package -->
229            <relocation>
230              <pattern>com.google.common</pattern>
231              <shadedPattern>myapp.shaded.com.google.common</shadedPattern>
232            </relocation>
233          </relocations>
234        </configuration>
235      </execution>
236    </executions>
237  </plugin>
238```
239