PermissionDenied
google.api_core.exceptions.PermissionDenied
Stack trace
google.api_core.exceptions.PermissionDenied: 403 Permission denied on resource at google.cloud.aiplatform_v1.services.endpoint_service.EndpointServiceClient.predict(EndpointServiceClient.java:123) at main.py:42
Why it happens
The service account credentials provided do not have the required IAM roles or permissions to perform the requested Vertex AI operation. This can happen if the service account is missing roles like 'Vertex AI User' or 'Service Account User', or if the credentials file is incorrect or revoked.
Detection
Check for PermissionDenied exceptions in your logs or catch google.api_core.exceptions.PermissionDenied in your code to identify missing or insufficient permissions before retrying.
Causes & fixes
Service account lacks the 'Vertex AI User' IAM role required to access Vertex AI resources.
Grant the 'Vertex AI User' role to the service account in the Google Cloud Console IAM page.
The service account key file used is invalid, revoked, or incorrectly referenced in the environment.
Ensure the service account JSON key file is valid, correctly referenced by the GOOGLE_APPLICATION_CREDENTIALS environment variable, and not expired or revoked.
The service account does not have 'Service Account User' role needed to impersonate or use the account.
Assign the 'Service Account User' role to the user or service account that is trying to use the credentials.
Code: broken vs fixed
from google.cloud import aiplatform
client = aiplatform.gapic.PredictionServiceClient()
response = client.predict(endpoint='projects/123/locations/us-central1/endpoints/456', instances=[{}]) # PermissionDenied error hereimport os
from google.cloud import aiplatform
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/path/to/valid-service-account.json' # Set valid credentials
client = aiplatform.gapic.PredictionServiceClient()
response = client.predict(endpoint='projects/123/locations/us-central1/endpoints/456', instances=[{}])
print(response)Workaround
Catch the PermissionDenied exception and log detailed error info; temporarily use a user account with sufficient permissions to continue development while fixing service account roles.
Prevention
Use least privilege principle by assigning only necessary IAM roles like 'Vertex AI User' to service accounts and validate credentials setup in CI/CD pipelines before deployment.