GitHub Actions to S3 Bucket Sync with OIDC Authentication

Overview

This guide demonstrates how to create a secure CI/CD pipeline that automatically syncs files from a GitHub repository to an Amazon S3 bucket using GitHub Actions with OpenID Connect (OIDC) authentication. This approach eliminates the need for long-lived AWS access keys, providing enhanced security through temporary credentials.

Architecture

GitHub Repository → GitHub Actions → AWS OIDC Provider → IAM Role → S3 Bucket

Prerequisites

• AWS CLI configured with appropriate permissions

• GitHub repository with admin access

• AWS account with permissions to create IAM roles and S3 buckets

Step 1: Understanding OIDC Authentication

What is OIDC?

OpenID Connect (OIDC) is an identity layer built on top of OAuth 2.0. GitHub Actions can use OIDC to request short-lived access tokens from AWS without storing long-lived credentials.

Benefits of OIDC over Access Keys

• Enhanced Security: No long-lived credentials stored in GitHub

• Automatic Rotation: Tokens are short-lived and automatically refreshed

• Fine-grained Control: Restrict access to specific repositories and branches

• Audit Trail: Better tracking of which workflows accessed which resources

Step 2: Create AWS OIDC Identity Provider

Check if OIDC Provider Exists

aws iam get-open-id-connect-provider \
    --open-id-connect-provider-arn arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com

Create OIDC Provider (if it doesn't exist)

aws iam create-open-id-connect-provider \
    --url https://token.actions.githubusercontent.com \
    --client-id-list sts.amazonaws.com \
    --thumbprint-list 3943r443bef1233fae435rwe54rf831e3780aea1 \
    --thumbprint-list 2ef3a8bef1233fae4wef35rwefe54rf831re3464

Update Thumbprint (if provider already exists)

aws iam update-open-id-connect-provider-thumbprint \
    --open-id-connect-provider-arn arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com \
    --thumbprint-list 3943r443bef1233fae435rwe54rf831e3780aea1 2ef3a8bef1233fae4wef35rwefe54rf831re3464

Step 3: Create IAM Role and Policies

Trust Policy (trust-policy.json)

This policy allows GitHub Actions from your specific repository to assume the IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                },
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:<GITHUB_ORG_NAME>/<GITHUB_REPO_NAME>:*"
                }
            }
        }
    ]
}

Permissions Policy (permissions-policy.json)

This policy grants the role specific S3 permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_NAME>",
                "arn:aws:s3:::<S3_BUCKET_NAME>/*"
            ]
        }
    ]
}

Create IAM Role

aws iam create-role \
    --role-name <AWS_ROLE_NAME> \
    --assume-role-policy-document file://trust-policy.json \
    --description "Role for GitHub Actions to sync properties files to S3"

Attach Permissions Policy

Step 4: Configure S3 Bucket

Create S3 Bucket

aws s3 mb s3://<S3_BUCKET_NAME> --region <AWS_REGION>

S3 Bucket Policy (bucket-policy.json)

This policy restricts write access to only the GitHub Actions role:

{
    "Version": "2012-10-17",
    "Id": "RestrictWriteAccessToGithubActionsCI",
    "Statement": [
        {
            "Sid": "ExplicitDenyS3WriteForAllExceptGithubActions",
            "Effect": "Deny",
            "Principal": "*",
            "Action": [
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::<S3_BUCKET_NAME>/*",
            "Condition": {
                "StringNotEquals": {
                    "aws:PrincipalArn": "arn:aws:iam::<ACCOUNT_ID>:role/<AWS_ROLE_NAME>"
                }
            }
        },
        {
            "Sid": "AllowGithubActionsWrite",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/<AWS_ROLE_NAME>"
            },
            "Action": [
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_NAME>",
                "arn:aws:s3:::<S3_BUCKET_NAME>/*"
            ]
        }
    ]
}

Apply Bucket Policy

aws s3api put-bucket-policy \
    --bucket <S3_BUCKET_NAME> \
    --policy file://bucket-policy.json

Enable Versioning (Optional)

aws s3api put-bucket-versioning \
    --bucket <S3_BUCKET_NAME> \
    --versioning-configuration Status=Enabled

Step 5: Create GitHub Actions Workflow

Workflow File (.github/workflows/release.yml)

name: Sync Properties Files to S3

on:
  push:
    branches:
      - master
    paths:
      - '*.data'
      - '**/*.data'
  workflow_dispatch:

permissions:
  id-token: write
  contents: read

jobs:
  deploy-to-s3:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<AWS_ROLE_NAME>
          aws-region: ap-south-1 #<AWS_REGION_NAME>
          role-session-name: GitHubActions-SessionName

      - name: Validate properties files
        run: |
          echo "Validating data files..."
          find . -name "*.data" -type f | while read file; do
            if [ ! -r "$file" ]; then
              echo "Error: Cannot read $file"
              exit 1
            fi
          done
          echo "All data files validated"

      - name: Sync to S3
        run: |
          aws s3 sync . s3://<S3_BUCKET_NAME>/ \
            --delete \
            --exclude "*" \
            --include "*.data" \
            --exclude ".git/*" \
            --exclude ".github/*"

      - name: Verify sync
        run: |
          LOCAL_COUNT=$(find . -name "*.data" -type f | wc -l)
          S3_COUNT=$(aws s3 ls s3://<S3_BUCKET_NAME>/ --recursive | grep "\.data$" | wc -l)

          echo "Local files: $LOCAL_COUNT"
          echo "S3 files: $S3_COUNT"

          if [ "$LOCAL_COUNT" -ne "$S3_COUNT" ]; then
            echo "File count mismatch!"
            exit 1
          fi
          echo "Sync verified successfully"

      - name: Generate summary
        run: |
          echo "## Deployment Summary" >> $GITHUB_STEP_SUMMARY
          echo "- **Files Synced:** $(find . -name "*.data" -type f | wc -l)" >> $GITHUB_STEP_SUMMARY
          echo "- **Commit:** ${{ github.sha }}" >> $GITHUB_STEP_SUMMARY

Step 6: Configure GitHub Repository

Create Production Environment

1. Navigate to your GitHub repository

2. Go to Settings → Environments

3. Click New environment

4. Name it production

5. Configure any required reviewers (optional)

6. Save the environment

Key Workflow Components

• permissions: Grants necessary OIDC permissions

• environment: References the GitHub environment for additional security

• aws-actions/configure-aws-credentials@v4: Handles OIDC authentication

• role-to-assume: Specifies the IAM role to assume

Step 7: Testing and Verification

Verify AWS Configuration

# Check role exists
aws iam get-role --role-name <AWS_ROLE_NAME>

# Check role permissions
aws iam get-role-policy --role-name <AWS_ROLE_NAME> --policy-name S3AccessPolicy

# Check bucket policy
aws s3api get-bucket-policy --bucket <S3_BUCKET_NAME>

Test the Pipeline

1. Make a change to a properties file

2. Commit and push to the master branch

3. Monitor the GitHub Actions tab

4. Verify files appear in the S3 bucket

Security Best Practices

1. Principle of Least Privilege

• Grant only necessary permissions to the IAM role

• Restrict access to specific S3 bucket and objects

2. Repository-Specific Access

• Use specific repository names in trust policy conditions

• Avoid wildcards unless necessary

3. Branch Protection

• Restrict role assumption to specific branches

• Use branch protection rules in GitHub

4. Environment Protection

• Use GitHub environments for additional approval workflows

• Configure required reviewers for sensitive deployments

5. Monitoring and Auditing

• Enable CloudTrail for AWS API calls

• Monitor GitHub Actions logs

• Set up alerts for failed deployments

Troubleshooting Common Issues

1. "Not authorized to perform sts:AssumeRoleWithWebIdentity"

Causes:

• Incorrect trust policy configuration

• Missing or incorrect OIDC provider

• Repository/branch name mismatch

Solutions:

• Verify trust policy repository name

• Check OIDC provider thumbprints

• Ensure branch names match

2. "Access Denied" during S3 operations

Causes:

• Insufficient IAM role permissions

• Incorrect S3 bucket policy

• Wrong bucket name or region

Solutions:

• Review IAM role permissions

• Check S3 bucket policy

• Verify bucket name and region

3. Workflow not triggering

Causes:

• Incorrect path filters

• Wrong branch name

• Missing workflow permissions

Solutions:

• Check workflow trigger conditions

• Verify branch names

• Ensure proper permissions are set

Cost Optimization

S3 Storage Classes

• Use appropriate storage classes for your use case

• Consider lifecycle policies for older versions

GitHub Actions Minutes

• Optimize workflow to run only when necessary

• Use path filters to avoid unnecessary runs

Conclusion

This setup provides a secure, automated way to sync files from GitHub to S3 using OIDC authentication. The approach eliminates the need for long-lived credentials while maintaining fine-grained access control. The pipeline automatically triggers on changes to properties files and provides comprehensive logging and verification.

Key Benefits Achieved

• Security: No long-lived AWS credentials stored in GitHub

• Automation: Automatic sync on code changes

• Reliability: Built-in validation and verification steps

• Auditability: Comprehensive logging and reporting

• Scalability: Easy to extend to multiple repositories or S3 buckets

This implementation serves as a foundation that can be extended for more complex deployment scenarios and additional security requirements.