Deployment Plan: Convert To Markdown - Complete Commercial Setup

Overview

This document outlines the complete deployment plan for Convert To Markdown, including both the open-source conversion service and the commercial subscription system. The architecture uses a two-repository approach to separate public and private code.

IMPORTANT REGIONAL NOTE: This deployment uses us-east4 (Northern Virginia) which supports free custom domain mapping. This region was chosen for its support of domain mapping, which saves ~$18/month in load balancer costs. See Step 13 for details on domain mapping benefits.

Architecture Overview

Repository Structure

public-repo/ (xlsx-docx-ppt-convert-to-md)
├── src/                    # Public conversion functions
├── lib/                    # Core conversion library
├── package.json           # NPM package
└── docs/                  # Public documentation

private-repo/ (convert-to-markdown-commercial)
├── functions/             # Commercial Cloud Functions
│   ├── stripe-webhook/
│   ├── api-validator/
│   └── customer-api/
├── infrastructure/        # Deployment configs
├── emails/               # Gmail integration
└── .env.production       # Secret configuration

Prerequisites

• Google Cloud account with billing enabled
• gcloud CLI installed and authenticated
• Node.js 20.x installed locally
• Stripe account created and configured
• Gmail API access configured
• Private repository created for commercial code
• Git installed and configured
• Appropriate GCP permissions (Project Creator, Billing Account User)

Automated Deployment Scripts

NEW: We've created automated deployment scripts to make this process easier!

Quick Start:

cd deploy-gcp
chmod +x *.sh

# Run scripts in order:
./00-setup-config.sh         # Create configuration
./01-setup-gcp-account.sh    # GCP authentication
./02-create-project.sh       # Create project
./03-enable-apis.sh          # Enable APIs
./04-configure-project.sh    # Configure settings
./05-setup-firestore.sh      # Set up database
./06-deploy-public-functions.sh  # Deploy functions
./07-test-public-functions.sh    # Test endpoints

# Commercial features (optional):
./08-setup-stripe.sh         # Configure payments
./09-setup-gmail.sh          # Set up email
./10-deploy-commercial-functions.sh  # Deploy commercial
./11-deploy-static-site.sh   # Deploy docs site
./12-test-commercial-functions.sh   # Test commercial

Configuration: All scripts use a centralized deployment.env file for configuration. Run ./00-setup-config.sh first to create this from a template.

Common Issue: If you get "command not found" errors, it's usually due to unquoted values with spaces in deployment.env. Use ./fix-deployment-env.sh to fix automatically, or ./validate-config.sh to check your configuration.

See deploy-gcp/README.md for detailed script documentation.

---

Manual Deployment Instructions

If you prefer to run commands manually, follow the steps below:

Step-by-Step Deployment Plan

Phase 0: Setup and Authentication

Step 0.0: Create Deployment Configuration

Script: ./00-setup-config.sh

This script creates a deployment.env file from a template where you configure all your project settings, API keys, and deployment preferences.

Manual setup:

# Copy template and edit configuration
cp config-template.env deployment.env
nano deployment.env  # Edit with your values

# Validate your configuration (recommended)
./validate-config.sh

# Fix common issues if needed
./fix-deployment-env.sh

Important: Values with spaces must be quoted:

• PROJECT_NAME="My Project"
• PROJECT_NAME=My Project

Step 0.1: Authenticate with Google Cloud

Script: ./01-setup-gcp-account.sh

Manual commands:

# Check current authenticated accounts
gcloud auth list

# If not authenticated or wrong account, login
gcloud auth login

# Select the correct account if multiple are available
gcloud config set account YOUR_EMAIL@DOMAIN.COM

Step 0.2: Verify Billing Account Access

Script: Included in ./01-setup-gcp-account.sh

Manual commands:

# List available billing accounts
gcloud billing accounts list

# Note the ACCOUNT_ID for the billing account you want to use
# Format: XXXXXX-XXXXXX-XXXXXX

Step 0.3: Verify Organization/Folder Permissions (if applicable)

# If creating project under an organization
gcloud organizations list

# If creating project under a folder
gcloud resource-manager folders list --organization=YOUR_ORG_ID

Phase 1: Core Infrastructure Setup

Step 1: Create New Google Cloud Project

Script: ./02-create-project.sh

Manual commands:

# Choose a unique project ID (must be globally unique)
PROJECT_ID="convert-to-markdown-prod"  # Change this to your preferred ID
PROJECT_NAME="Convert To Markdown"
BILLING_ACCOUNT_ID="XXXXXX-XXXXXX-XXXXXX"  # Replace with your billing account ID from Step 0.2

# Create new project
echo "Creating project: $PROJECT_ID"
gcloud projects create $PROJECT_ID \
  --name="$PROJECT_NAME" \
  --labels=environment=production,product=convert-to-markdown

# Verify project was created
gcloud projects describe $PROJECT_ID

# Set the new project as active
gcloud config set project $PROJECT_ID

# Verify current project
echo "Current project:"
gcloud config get-value project

# Link billing account
echo "Linking billing account..."
gcloud beta billing projects link $PROJECT_ID \
  --billing-account=$BILLING_ACCOUNT_ID

# Verify billing is enabled
gcloud billing projects describe $PROJECT_ID

Step 2: Enable Required APIs

Script: ./03-enable-apis.sh

Manual commands:

# Enable necessary APIs for the new project
echo "Enabling required APIs..."

# Core APIs
gcloud services enable cloudfunctions.googleapis.com \
  run.googleapis.com \
  cloudbuild.googleapis.com \
  storage.googleapis.com \
  cloudresourcemanager.googleapis.com \
  compute.googleapis.com \
  --project=$PROJECT_ID

# Wait for APIs to propagate
echo "Waiting for APIs to be fully enabled..."
sleep 30

# Database and commercial features
gcloud services enable firestore.googleapis.com \
  gmail.googleapis.com \
  cloudscheduler.googleapis.com \
  --project=$PROJECT_ID

# Additional APIs that might be needed
gcloud services enable \
  logging.googleapis.com \
  monitoring.googleapis.com \
  cloudtasks.googleapis.com \
  --project=$PROJECT_ID

# Verify APIs are enabled
echo "Verifying enabled APIs..."
gcloud services list --enabled --project=$PROJECT_ID | grep -E "(functions|firestore|gmail|storage)"

# Detailed verification of each critical API
echo
echo "Detailed API verification:"
for api in cloudfunctions firestore gmail storage cloudbuild compute; do
  if gcloud services list --enabled --project=$PROJECT_ID | grep -q "$api"; then
    echo " $api API is enabled"
  else
    echo " ERROR: $api API is NOT enabled"
    exit 1
  fi
done

Step 3: Set Default Configuration

Script: ./04-configure-project.sh

Manual commands:

# Set default region
# IMPORTANT: This deployment uses us-east4 for free domain mapping
# See Step 13 for details on regional requirements and alternatives
REGION="us-east4"  # Northern Virginia (supports free domain mapping)

# Set project-specific configuration
gcloud config set project $PROJECT_ID
gcloud config set functions/region $REGION
gcloud config set run/region $REGION
gcloud config set compute/region $REGION

# Create a configuration for this project (optional but recommended)
gcloud config configurations create convert-to-markdown-prod
gcloud config set project $PROJECT_ID
gcloud config set functions/region $REGION
gcloud config set account YOUR_EMAIL@DOMAIN.COM

# Verify configuration
echo "Current configuration:"
gcloud config list
gcloud config configurations list

Step 4: Set Up Firestore Database

Script: ./05-setup-firestore.sh

Manual commands:

# Check if Firestore database already exists
echo "Checking for existing Firestore database..."
if gcloud firestore databases list --project=$PROJECT_ID 2>/dev/null | grep -q "(default)"; then
  echo " Firestore database already exists"
else
  echo "Creating new Firestore database..."
  
  # Create Firestore database in Native mode
  gcloud firestore databases create \
    --location=$REGION \
    --project=$PROJECT_ID
  
  # Wait for database to be ready
  echo "Waiting for Firestore to initialize..."
  sleep 30
  
  # Verify database creation
  if gcloud firestore databases list --project=$PROJECT_ID 2>/dev/null | grep -q "(default)"; then
    echo " Firestore database created successfully"
  else
    echo " ERROR: Failed to create Firestore database"
    exit 1
  fi
fi

# Create Firestore indexes
echo "Creating Firestore indexes..."
echo "Note: Indexes will be created individually using gcloud commands"

# Create indexes programmatically
echo "Creating user email index..."
gcloud firestore indexes composite create \
  --collection-group=users \
  --field-config=field-path=email,order=ASCENDING \
  --field-config=field-path=createdAt,order=DESCENDING \
  --project=$PROJECT_ID \
  --async || echo "Index may already exist"

echo "Creating API keys index..."
gcloud firestore indexes composite create \
  --collection-group=apiKeys \
  --field-config=field-path=userId,order=ASCENDING \
  --field-config=field-path=isActive,order=ASCENDING \
  --project=$PROJECT_ID \
  --async || echo "Index may already exist"

echo "Creating usage index..."
gcloud firestore indexes composite create \
  --collection-group=usage \
  --field-config=field-path=userId,order=ASCENDING \
  --field-config=field-path=timestamp,order=DESCENDING \
  --project=$PROJECT_ID \
  --async || echo "Index may already exist"

# Optional: Create firestore.indexes.json for Firebase CLI (future use)
echo
echo "Creating firestore.indexes.json for future Firebase CLI deployments..."
cat > firestore.indexes.json << 'EOF'
{
  "indexes": [
    {
      "collectionGroup": "users",
      "queryScope": "COLLECTION",
      "fields": [
        { "fieldPath": "email", "order": "ASCENDING" },
        { "fieldPath": "createdAt", "order": "DESCENDING" }
      ]
    },
    {
      "collectionGroup": "apiKeys",
      "queryScope": "COLLECTION",
      "fields": [
        { "fieldPath": "userId", "order": "ASCENDING" },
        { "fieldPath": "isActive", "order": "ASCENDING" }
      ]
    },
    {
      "collectionGroup": "usage",
      "queryScope": "COLLECTION", 
      "fields": [
        { "fieldPath": "userId", "order": "ASCENDING" },
        { "fieldPath": "timestamp", "order": "DESCENDING" }
      ]
    }
  ],
  "fieldOverrides": []
}
EOF

echo "Indexes created! Note: They may take a few minutes to build."
echo "To check index status, visit:"
echo "https://console.cloud.google.com/firestore/indexes?project=$PROJECT_ID"
echo
echo "To verify operations:"
gcloud firestore operations list --project=$PROJECT_ID --limit=5

Phase 2: Public Service Deployment

Step 5: Prepare for Deployment

Script: Included in ./06-deploy-public-functions.sh

Manual commands:

# Clone the public repository if not already present
if [ ! -d "xlsx-docx-ppt-convert-to-md" ]; then
  echo "Cloning public repository..."
  git clone https://github.com/your-org/xlsx-docx-ppt-convert-to-md.git
  cd xlsx-docx-ppt-convert-to-md
else
  echo "Using existing repository..."
  cd xlsx-docx-ppt-convert-to-md
  git pull origin main
fi

# Install dependencies
echo "Installing dependencies..."
npm install

# Create deployment directory
mkdir -p deployment-scripts
cd deployment-scripts

# Create a deployment environment file
cat > deployment.env << EOF
# Deployment Configuration
PROJECT_ID=$PROJECT_ID
REGION=$REGION
DEPLOYMENT_DATE=$(date +%Y-%m-%d)
DEPLOYMENT_VERSION=$(git rev-parse --short HEAD)
EOF

# Verify we're in the correct project
echo
echo "==========================================="
echo "Deployment Pre-flight Check"
echo "==========================================="
echo "Current project: $(gcloud config get-value project)"
echo "Expected project: $PROJECT_ID"
echo "Region: $REGION"
echo "Git commit: $(git rev-parse --short HEAD)"
echo "==========================================="

# Confirm before proceeding
read -p "Is this the correct configuration? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
    echo "Deployment cancelled. Please verify configuration."
    exit 1
fi

# Test that source files exist
echo "Verifying source files..."
required_files=(
  "../src/xlsxConverter.js"
  "../src/pdfConverter.js"
  "../src/docxConverter.js"
  "../package.json"
)

for file in "${required_files[@]}"; do
  if [ -f "$file" ]; then
    echo " Found: $file"
  else
    echo " Missing: $file"
    echo "ERROR: Required source files not found. Please check your repository structure."
    exit 1
  fi
done

Step 6: Deploy Public Conversion Functions

Script: ./06-deploy-public-functions.sh

Manual deployment script:

#!/bin/bash

# Exit on error
set -e

# Use environment variables or prompt for values
if [ -z "$PROJECT_ID" ]; then
    read -p "Enter GCP Project ID: " PROJECT_ID
fi

if [ -z "$REGION" ]; then
    read -p "Enter GCP Region (default: us-east4 for free domain mapping): " REGION
    REGION=${REGION:-us-east4}
fi

echo "=========================================="
echo "Deployment Configuration:"
echo "Project ID: $PROJECT_ID"
echo "Region: $REGION"
echo "Current Account: $(gcloud config get-value account)"
echo "=========================================="

# Confirm deployment
read -p "Proceed with deployment? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
    echo "Deployment cancelled."
    exit 1
fi

# Ensure we're using the correct project
gcloud config set project $PROJECT_ID

# Verify the source files exist
if [ ! -f "../src/xlsxConverter.js" ]; then
    echo "ERROR: Source files not found. Please run this script from the deployment-scripts directory."
    echo "Current directory: $(pwd)"
    exit 1
fi

# Deploy Excel (.xlsx, .xls, .xlsm) to JSON converter
echo "Deploying xlsx-converter..."
gcloud functions deploy xlsx-converter \
  --gen2 \
  --runtime=nodejs20 \
  --region=$REGION \
  --source=.. \
  --entry-point=excelToJson \
  --trigger-http \
  --allow-unauthenticated \
  --memory=512MB \
  --timeout=60s \
  --max-instances=100 \
  --project=$PROJECT_ID

# Deploy Excel (.xlsx, .xls, .xlsm) to Markdown converter
echo "Deploying xlsx-to-md..."
gcloud functions deploy xlsx-to-md \
  --gen2 \
  --runtime=nodejs20 \
  --region=$REGION \
  --source=.. \
  --entry-point=excelToMarkdown \
  --trigger-http \
  --allow-unauthenticated \
  --memory=512MB \
  --timeout=60s \
  --max-instances=100 \
  --project=$PROJECT_ID

# Deploy PDF to Markdown converter
echo "Deploying pdf-to-md..."
gcloud functions deploy pdf-to-md \
  --gen2 \
  --runtime=nodejs20 \
  --region=$REGION \
  --source=.. \
  --entry-point=pdfToMarkdown \
  --trigger-http \
  --allow-unauthenticated \
  --memory=512MB \
  --timeout=60s \
  --max-instances=100 \
  --project=$PROJECT_ID

# Deploy Word (.docx, .dotx, .dotm) to HTML converter
echo "Deploying docx-to-html..."
gcloud functions deploy docx-to-html \
  --gen2 \
  --runtime=nodejs20 \
  --region=$REGION \
  --source=.. \
  --entry-point=docxToHtml \
  --trigger-http \
  --allow-unauthenticated \
  --memory=512MB \
  --timeout=60s \
  --max-instances=100 \
  --project=$PROJECT_ID

# Deploy Word (.docx, .dotx, .dotm) to Markdown converter
echo "Deploying docx-to-md..."
gcloud functions deploy docx-to-md \
  --gen2 \
  --runtime=nodejs20 \
  --region=$REGION \
  --source=.. \
  --entry-point=docxToMarkdown \
  --trigger-http \
  --allow-unauthenticated \
  --memory=512MB \
  --timeout=60s \
  --max-instances=100 \
  --project=$PROJECT_ID

echo "All public functions deployed successfully!"
echo
echo "Getting function URLs..."

# List all deployed functions with their URLs
gcloud functions list --project=$PROJECT_ID --format="table(name,httpsTrigger.url)"

# Save URLs to file
echo
echo "Saving function URLs to function-urls.txt..."
gcloud functions list --project=$PROJECT_ID --format="value(name,httpsTrigger.url)" > function-urls.txt

# Test each function endpoint
echo
echo "Testing function endpoints..."
echo "Creating test files..."

# Create minimal test files
echo "Test,Data" > test.csv
echo "Test content" > test.txt

# Test each endpoint
base_url="https://$REGION-$PROJECT_ID.cloudfunctions.net"

echo "Testing xlsx-converter (expecting 400 for CSV file)..."
curl -s -o /dev/null -w "%{http_code}" -X POST "$base_url/xlsx-converter" -F "file=@test.csv"
echo

echo "Testing pdf-to-md (expecting 400 for TXT file)..."
curl -s -o /dev/null -w "%{http_code}" -X POST "$base_url/pdf-to-md" -F "file=@test.txt"
echo

# Clean up test files
rm test.csv test.txt

echo
echo "Deployment complete! Function URLs saved to function-urls.txt"
echo "Next steps:"
echo "1. Test each function with proper file types"
echo "2. Update your documentation with the new URLs"
echo "3. Set up monitoring and alerts"

Phase 3: Commercial System Setup

Step 7: Configure Stripe Integration

Script: ./08-setup-stripe.sh (runs scripts in deploy-stripe/ directory)

The Stripe setup is now automated using the Stripe CLI and API. The script will create a two-tier pricing model:

• Free Tier: $0/month for 50 conversions
• Pro Tier: $10/month for unlimited conversions

7.1: Prerequisites

1. Install Stripe CLI:

   # The script will offer to install for you, or run manually:
   ./deploy-stripe/install-stripe-cli.sh
   
   # macOS: Uses Homebrew
   # Linux: Downloads binary from GitHub
   # Windows: Provides instructions
   

2. Get Stripe API Keys:

   • Go to https://dashboard.stripe.com/apikeys
   • Copy your Publishable key (starts with pk_test_ or pk_live_)
   • Copy your Secret key (starts with sk_test_ or sk_live_)

7.2: Run Automated Setup

# From deploy-gcp directory
./08-setup-stripe.sh

# This will:
# 1. Check/install Stripe CLI
# 2. Authenticate with your Stripe account
# 3. Create/update product and prices
# 4. Configure customer portal
# 5. Set up webhook endpoints
# 6. Save all configuration

7.3: What the Script Creates

1. Product Configuration:

   • Name: "Convert to Markdown API"
   • Description: Professional document conversion API
   • Metadata: Free conversions limit (50)

2. Price Tiers:

   # Free Tier
   - Price: $0/month
   - ID: price_xxxxx (auto-generated)
   - Metadata: tier=free, conversions_limit=50
   
   # Pro Tier  
   - Price: $10/month
   - ID: price_1Ro0PGAic9M7TwKd81k9YDLt (or new if not exists)
   - Metadata: tier=pro, conversions_limit=unlimited
   

3. Customer Portal:

   • Allows customers to manage subscriptions
   • Update payment methods
   • Cancel subscriptions
   • View invoices
   • Update billing address

4. Webhook Configuration:

   # Webhook URL: https://REGION-PROJECT_ID.cloudfunctions.net/stripe-webhook
   # Events monitored:
   - checkout.session.completed
   - customer.subscription.created/updated/deleted
   - invoice.payment_succeeded/failed
   

7.4: Manual Step - Create Pricing Table

The Stripe CLI doesn't support creating pricing tables, so this must be done manually:

1. Navigate to Pricing Tables:

   • Go to https://dashboard.stripe.com/pricing-tables
   • Click + Create pricing table

2. Add Products to Table:

   Free Tier (use the price created by script):

   Title: Free
   Price: $0/month
   Product: Select the free tier price created by the script
   Features:
   - 50 conversions per month
   - 5MB file size limit
   - All file formats
   - Community support
   Button text: Get Started
   

   Pro Plan (use existing or script-created price):

   Title: Pro
   Price: $10/month
   Product: Select the pro tier price (price_1Ro0PGAic9M7TwKd81k9YDLt)
   Features:
   - 10,000 conversions per month
   - 50MB file size limit
   - All file formats
   - API access with authentication
   - Priority email support
   - Usage dashboard
   - Monthly usage reset
   Button text: Subscribe
   Highlighted: Yes 
   

3. Configure Table Settings:

   Header: Simple, Predictable Pricing
   Layout: 2 columns
   Show feature comparison: Yes
   Currency: USD
   

4. Get Embed Code:

   • Click Copy embed code
   • You'll get code like:

   <script async src="https://js.stripe.com/v3/pricing-table.js"></script>
   <stripe-pricing-table 
     pricing-table-id="prctbl_XXXXXXXXXXXXX"
     publishable-key="pk_live_XXXXXXXXXXXXX">
   </stripe-pricing-table>
   

   • Save the pricing-table-id to your deployment.env as STRIPE_PRICING_TABLE_ID

7.5: Environment Variables Created

The script automatically updates your deployment.env with:

STRIPE_PUBLISHABLE_KEY=pk_test_xxxxx or pk_live_xxxxx
STRIPE_SECRET_KEY=sk_test_xxxxx or sk_live_xxxxx
STRIPE_PRODUCT_ID=prod_xxxxx
STRIPE_PRICE_ID=price_xxxxx (pro tier)
STRIPE_FREE_PRICE_ID=price_xxxxx (free tier)
STRIPE_PORTAL_CONFIG_ID=bpc_xxxxx
STRIPE_WEBHOOK_ENDPOINT_ID=we_xxxxx
STRIPE_WEBHOOK_SECRET=whsec_xxxxx

Also creates .env.production with all necessary configuration for commercial functions.

7.6: Testing Your Setup

1. Test Webhook Locally:

   # In one terminal, forward webhooks to your local server
   stripe listen --forward-to localhost:8080
   
   # In another terminal, trigger test events
   stripe trigger payment_intent.succeeded
   stripe trigger customer.subscription.created
   

2. Test Card Numbers:

   Success: 4242 4242 4242 4242
   Decline: 4000 0000 0000 0002
   3D Secure: 4000 0025 0000 3155
   

3. Verify Customer Portal:

   • The portal URL format is: https://billing.stripe.com/p/login/test_xxxxx
   • Find your portal configuration ID in Stripe Dashboard or deployment.env

7.7: Stripe Setup Directory Structure

The Stripe setup is organized in the deploy-stripe/ subdirectory:

deploy-gcp/
├── 08-setup-stripe.sh          # Wrapper script
└── deploy-stripe/
    ├── README.md               # Detailed Stripe documentation
    ├── install-stripe-cli.sh   # CLI installation script
    ├── stripe-helpers.sh       # Helper functions
    └── setup-stripe.sh         # Main setup script

7.8: Implementation Details

The automated setup creates the foundation for subscription management. Your commercial functions will need to:

1. Handle Webhook Events (stripe-webhook function):

   • Process subscription lifecycle events
   • Update user records in Firestore
   • Track payment status

2. Manage Usage (in your conversion functions):

   • Check user's subscription status
   • Track conversions against limits (50 free, 10000 pro)
   • Reset usage monthly via Cloud Scheduler

3. Firestore Structure:

   subscriptions/
   ├── {userId}/
   │   ├── stripeCustomerId: "cus_XXXXX"
   │   ├── subscriptionId: "sub_XXXXX"
   │   ├── status: "active|canceled|past_due"
   │   ├── plan: "free|pro"
   │   └── monthlyUsage: 0
   

7.9: Testing Checklist

• Test checkout flow with test card
• Verify webhook receives events
• Confirm subscription created in Firestore
• Test usage tracking increments
• Verify usage limit enforcement
• Test customer portal access
• Confirm monthly usage reset
• Test subscription cancellation
• Verify email notifications sent

7.10: Go Live Checklist

• Switch Stripe to Live mode
• Update all API keys to live versions
• Test with real payment method ($10)
• Verify webhook signature validation
• Confirm usage resets on the 1st
• Monitor first real customer signup
• Set up Stripe email notifications

7.11: Troubleshooting Stripe Setup

Common Issues:

1. "Stripe CLI not found":

   # Run the installation script
   ./deploy-stripe/install-stripe-cli.sh
   

2. "Invalid API key":

   • Ensure you're using the correct mode (test vs live)
   • Check that keys are properly quoted in deployment.env

3. "Product already exists":

   • The script is idempotent and will update existing products
   • Check Stripe Dashboard for duplicate products

4. "Webhook signature verification failed":

   • Ensure STRIPE_WEBHOOK_SECRET matches the one from Stripe Dashboard
   • Check that the webhook URL is correct in Stripe settings

For more details, see deploy-stripe/README.md.

Step 8: Set Up Gmail API

Script: ./09-setup-gmail.sh

Manual commands:

# Create service account for Gmail
echo "Creating Gmail service account..."
gcloud iam service-accounts create gmail-sender \
  --display-name="Gmail Email Sender" \
  --description="Service account for sending transactional emails" \
  --project=$PROJECT_ID

# Create keys directory
mkdir -p ../keys

# Download service account key
echo "Downloading service account key..."
gcloud iam service-accounts keys create ../keys/gmail-service-account.json \
  --iam-account=gmail-sender@$PROJECT_ID.iam.gserviceaccount.com \
  --project=$PROJECT_ID

echo "Service account created: gmail-sender@$PROJECT_ID.iam.gserviceaccount.com"
echo "Key saved to: ../keys/gmail-service-account.json"
echo
echo "IMPORTANT: For Gmail API to work, you need to:"
echo "1. Enable domain-wide delegation in Google Workspace admin"
echo "2. Add the service account email to authorized senders"
echo "3. Grant 'https://www.googleapis.com/auth/gmail.send' scope"

Step 9: Create Environment Configuration

# Create environment file template
cat > ../.env.production.template << 'EOF'
# Stripe Configuration
STRIPE_PUBLISHABLE_KEY=pk_live_xxx
STRIPE_SECRET_KEY=sk_live_xxx
STRIPE_WEBHOOK_SECRET=whsec_xxx
STRIPE_PRICING_TABLE_ID=prctbl_xxx

# Google Cloud
GCP_PROJECT_ID=${PROJECT_ID}
FIRESTORE_DATABASE=production
GMAIL_SERVICE_ACCOUNT=./keys/gmail-service-account.json

# Email Configuration
ADMIN_EMAIL=lindsay@knowcode.tech
SUPPORT_EMAIL=lindsay@knowcode.tech
FROM_EMAIL=noreply@knowcode.tech

# API Configuration
API_DOMAIN=https://api.convert-to-markdown.com
WEBSITE_DOMAIN=https://convert-to-markdown.com

# Security
API_KEY_PREFIX_PROD=ctm_live_
API_KEY_PREFIX_TEST=ctm_test_
EOF

echo "Environment template created: ../.env.production.template"
echo "Copy this to .env.production and fill in the actual values"

Step 10: Deploy Commercial Functions

Script: ./10-deploy-commercial-functions.sh

Manual deployment script:

#!/bin/bash

set -e

# Load environment variables
if [ -f ".env.production" ]; then
  export $(cat .env.production | grep -v '^#' | xargs)
fi

PROJECT_ID="${PROJECT_ID:-doc-converter-prod-v2}"
REGION="${REGION:-us-east4}"  # Using us-east4 for free domain mapping
PRIVATE_REPO_PATH="../convert-to-markdown-commercial"

# Verify private repository exists
if [ ! -d "$PRIVATE_REPO_PATH" ]; then
  echo "ERROR: Private repository not found at $PRIVATE_REPO_PATH"
  echo "Please clone the commercial repository first."
  exit 1
fi

# Verify required environment variables
required_vars=(
  "STRIPE_SECRET_KEY"
  "STRIPE_WEBHOOK_SECRET"
)

for var in "${required_vars[@]}"; do
  if [ -z "${!var}" ]; then
    echo "ERROR: Required environment variable $var is not set"
    echo "Please check your .env.production file"
    exit 1
  fi
done

echo "Deploying commercial functions to project: $PROJECT_ID"
echo "Using private repository: $PRIVATE_REPO_PATH"

# Deploy Stripe webhook handler
echo "Deploying stripe-webhook..."
gcloud functions deploy stripe-webhook \
  --gen2 \
  --runtime=nodejs20 \
  --region=$REGION \
  --source=$PRIVATE_REPO_PATH/functions/stripe-webhook \
  --entry-point=stripeWebhook \
  --trigger-http \
  --allow-unauthenticated \
  --set-env-vars="STRIPE_SECRET_KEY=$STRIPE_SECRET_KEY,STRIPE_WEBHOOK_SECRET=$STRIPE_WEBHOOK_SECRET" \
  --set-labels="type=commercial" \
  --project=$PROJECT_ID

# Deploy API key validator
echo "Deploying api-key-validator..."
gcloud functions deploy api-key-validator \
  --gen2 \
  --runtime=nodejs20 \
  --region=$REGION \
  --source=$PRIVATE_REPO_PATH/functions/api-validator \
  --entry-point=validateApiKey \
  --trigger-http \
  --allow-unauthenticated \
  --set-labels="type=commercial" \
  --project=$PROJECT_ID

# Deploy commercial API endpoints (authenticated versions)
for converter in xlsx-converter pdf-to-md docx-to-html docx-to-md xlsx-to-md; do
  echo "Deploying commercial-$converter..."
  gcloud functions deploy commercial-$converter \
    --gen2 \
    --runtime=nodejs20 \
    --region=$REGION \
    --source=$PRIVATE_REPO_PATH/functions/commercial-wrapper \
    --entry-point=commercial${converter//-/} \
    --trigger-http \
    --allow-unauthenticated \
    --set-env-vars="PUBLIC_PACKAGE=@knowcode/convert-to-markdown" \
    --set-labels="type=commercial" \
    --project=$PROJECT_ID
done

# Deploy customer API
echo "Deploying customer-api..."
gcloud functions deploy customer-api \
  --gen2 \
  --runtime=nodejs20 \
  --region=$REGION \
  --source=$PRIVATE_REPO_PATH/functions/customer-api \
  --entry-point=customerApi \
  --trigger-http \
  --allow-unauthenticated \
  --set-labels="type=commercial" \
  --project=$PROJECT_ID

echo "Commercial functions deployed!"

# Verify deployment
echo
echo "Verifying commercial function deployments..."
commercial_functions=(
  "stripe-webhook"
  "api-key-validator"
  "commercial-xlsx-converter"
  "commercial-pdf-to-md"
  "commercial-docx-to-html"
  "commercial-docx-to-md"
  "commercial-xlsx-to-md"
  "customer-api"
)

for func in "${commercial_functions[@]}"; do
  if gcloud functions describe $func --region=$REGION --project=$PROJECT_ID &>/dev/null; then
    echo " $func deployed successfully"
  else
    echo " $func deployment failed"
  fi
done

Phase 4: Static Site Updates

Script: ./11-deploy-static-site.sh

Step 11: Update .gcloudignore

Ensure .gcloudignore excludes unnecessary files:

.git
.gitignore
node_modules
test/
docs/
html/
recordings/
*.md
.DS_Store
npm-debug.log
.env
.vscode/

Step 11: Create Pricing Page

The pricing page should reflect the new $10/month flat-rate pricing model. Update or create pricing.md in docs folder with the pricing content that emphasizes predictable pricing and includes the Stripe pricing table embed.

Key elements to include:

• $10/month flat rate as the main feature
• 10,000 conversions per month for Pro plan
• 50 conversions per month for Free tier with 5MB limit
• Stripe pricing table embed code
• Clear comparison between Free and Pro tiers
• Enterprise contact option for higher volumes

The pricing table embed will use the IDs from Step 7.4:

<stripe-pricing-table 
  pricing-table-id="${STRIPE_PRICING_TABLE_ID}"
  publishable-key="${STRIPE_PUBLISHABLE_KEY}">
</stripe-pricing-table>

Step 12: Deploy Everything

# Make deployment script executable
chmod +x deploy-new-project.sh

# Run deployment
./deploy-new-project.sh

Phase 5: Post-Deployment Configuration

Step 13: Configure Custom Domain COMPLETED

Status: SSL configuration and domain mapping are fully configured and active.

13.1: Current Domain Configuration

Active Setup:

• Domain: convert-to-markdown.knowcode.tech
• SSL Certificate: Auto-provisioned and managed by Google Cloud
• Region: us-east4 (Northern Virginia)
• Cost: $0/month (free domain mapping, no load balancer required)
• Pricing Table ID: prctbl_1Ro4D6Aic9M7TwKdm0DJZhGF

Accessible Function URLs:

https://convert-to-markdown.knowcode.tech/xlsx-converter
https://convert-to-markdown.knowcode.tech/xlsx-to-md
https://convert-to-markdown.knowcode.tech/pdf-to-md
https://convert-to-markdown.knowcode.tech/docx-to-html
https://convert-to-markdown.knowcode.tech/docx-to-md

13.2: Benefits of This Configuration

Zero monthly infrastructure cost - No load balancer fees
Automatic SSL certificate renewal - Google manages certificates
Global edge caching - Fast worldwide access via Google's network
Simple management - No complex routing rules or manual SSL updates

13.3: Verification Commands

To verify your domain and SSL configuration:

# Check domain mapping status
gcloud beta run domain-mappings list --region=us-east4

# Test SSL certificate
curl -I https://convert-to-markdown.knowcode.tech

# Verify function accessibility
curl -X POST https://convert-to-markdown.knowcode.tech/xlsx-converter

# Check SSL certificate details
echo | openssl s_client -servername convert-to-markdown.knowcode.tech -connect convert-to-markdown.knowcode.tech:443 2>/dev/null | openssl x509 -noout -text | grep -A2 "Subject:"

13.4: Monitoring SSL Certificate

Google Cloud automatically handles SSL certificate renewal. To monitor:

# View certificate expiration
gcloud compute ssl-certificates list --project=$PROJECT_ID

# The certificate auto-renews ~30 days before expiration

13.5: Troubleshooting

If you encounter issues:

1. DNS Issues: Ensure DNS records point to Google's IPs
2. SSL Errors: Wait 10-15 minutes for provisioning
3. Function Access: Verify functions are deployed to us-east4

For support: Check Cloud Run domain mapping docs

Step 14: Configure Stripe Webhook (Already Covered)

This step is now covered in detail in Step 7.5. The webhook should already be configured with all necessary events for the $10/month subscription model.

Quick Verification:

# Test webhook endpoint is accessible
curl -I https://us-east4-$PROJECT_ID.cloudfunctions.net/stripe-webhook

# Should return 405 (Method Not Allowed) for GET request
# This confirms the endpoint exists

Webhook Events Summary:

• Checkout events for subscription creation
• Subscription lifecycle events
• Invoice events for payment tracking
• Customer events for user management

Step 15: Set Up Cloud Scheduler

# Enable Cloud Scheduler API if not already enabled
gcloud services enable cloudscheduler.googleapis.com --project=$PROJECT_ID

# Wait for API to be ready
sleep 10

# Create App Engine app (required for Cloud Scheduler)
echo "Creating App Engine app (required for Cloud Scheduler)..."
gcloud app create --region=$REGION --project=$PROJECT_ID || echo "App Engine app already exists"

# Create monthly usage reset job
echo "Creating monthly usage reset job..."
gcloud scheduler jobs create http reset-usage \
  --location=$REGION \
  --schedule="0 0 1 * *" \
  --uri="https://us-east4-$PROJECT_ID.cloudfunctions.net/reset-usage" \
  --http-method=POST \
  --time-zone="UTC" \
  --attempt-deadline="30m" \
  --headers="X-CloudScheduler-Auth=true" \
  --message-body='{"action":"reset_monthly_usage"}' \
  --project=$PROJECT_ID

# This job will:
# - Run at midnight UTC on the 1st of each month
# - Reset monthlyUsage to 0 for all subscriptions
# - Maintain subscription status and other fields

# Verify job creation
echo "Verifying scheduler job..."
gcloud scheduler jobs list --location=$REGION --project=$PROJECT_ID

Step 16: Document API Endpoints

Create api-endpoints.md:

# Production API Endpoints

## Public Endpoints (No Authentication)

Base URL: `https://us-east4-doc-converter-prod-v2.cloudfunctions.net`

### Demo/Test Endpoints
- `POST /xlsx-converter` - Limited to 3 requests per IP per hour
- `POST /pdf-to-md` - Limited to 3 requests per IP per hour

## Commercial Endpoints (API Key Required)

Base URL: `https://api.convert-to-markdown.com/v1`

### Conversion Endpoints
- `POST /convert/xlsx-to-json` - Excel (.xlsx, .xls, .xlsm) to JSON (authenticated)
- `POST /convert/xlsx-to-md` - Excel (.xlsx, .xls, .xlsm) to Markdown (authenticated)
- `POST /convert/pdf-to-md` - PDF to Markdown (authenticated)
- `POST /convert/docx-to-html` - Word (.docx, .dotx, .dotm) to HTML (authenticated)
- `POST /convert/docx-to-md` - Word (.docx, .dotx, .dotm) to Markdown (authenticated)

### Management Endpoints
- `GET /account/usage` - Get current usage statistics
- `GET /account/api-keys` - List API keys
- `POST /account/api-keys` - Create new API key
- `DELETE /account/api-keys/:id` - Revoke API key
- `GET /account/billing-portal` - Get Stripe portal URL

### Headers Required

x-api-key: ctm_live_xxxxxxxxxxxxx
Content-Type: multipart/form-data (for file uploads)

Phase 6: Testing & Validation

Step 17: Test Public Endpoints

Script: ./07-test-public-functions.sh

Manual test script:

#!/bin/bash

# Test script for new deployment
if [ -z "$PROJECT_ID" ]; then
    echo "ERROR: PROJECT_ID not set"
    exit 1
fi

if [ -z "$REGION" ]; then
    REGION="us-east4"  # Current deployment region
fi

BASE_URL="https://$REGION-$PROJECT_ID.cloudfunctions.net"

# Create test directory and files if they don't exist
mkdir -p test-files
cd test-files

# Create a simple Excel file using CSV (for testing)
cat > test.csv << EOF
Name,Value,Formula
Item A,100,=B2*2
Item B,200,=B3*2
Total,300,=SUM(B2:B3)
EOF

# Create a simple text file (for PDF testing)
cat > test.txt << EOF
TEST DOCUMENT

This is a test document for conversion.

Section 1: Introduction
This document tests the PDF to Markdown conversion.

Section 2: Data
Name    Age    City
John    25     NYC
Jane    30     LA
EOF

echo "Testing public endpoints at: $BASE_URL"
echo

# Test Excel converter (with CSV file - should fail gracefully)
echo "1. Testing xlsx-converter with CSV (should return error)..."
response=$(curl -s -w "\n\nHTTP_STATUS:%{http_code}" -X POST "$BASE_URL/xlsx-converter" -F "file=@test.csv")
http_status=$(echo "$response" | grep "HTTP_STATUS:" | cut -d':' -f2)
echo "   Status: $http_status"
if [ "$http_status" = "400" ] || [ "$http_status" = "415" ]; then
  echo "    Correctly rejected non-Excel file"
else
  echo "    Unexpected response for CSV file"
fi

# Test PDF converter (with text file - should process or fail gracefully)
echo
echo "2. Testing pdf-to-md with text file..."
response=$(curl -s -w "\n\nHTTP_STATUS:%{http_code}" -X POST "$BASE_URL/pdf-to-md" -F "file=@test.txt")
http_status=$(echo "$response" | grep "HTTP_STATUS:" | cut -d':' -f2)
echo "   Status: $http_status"

# Test rate limiting
echo
echo "3. Testing rate limiting (if implemented)..."
for i in {1..5}; do
  echo "   Request $i:"
  curl -s -o /dev/null -w "   HTTP Status: %{http_code}\n" \
    -X POST "$BASE_URL/xlsx-converter" \
    -F "file=@test.csv"
  sleep 1
done

# Clean up
cd ..
rm -rf test-files

echo
echo "Basic endpoint testing complete."
echo "For comprehensive testing, use actual Excel, PDF, and Word files."

Step 18: Test Commercial Flow

Script: ./12-test-commercial-functions.sh

Manual test commands:

#!/bin/bash

# Create test Stripe event
cat > stripe-test-event.json << 'EOF'
{
  "id": "evt_test_webhook",
  "object": "event",
  "api_version": "2023-10-16",
  "created": 1680000000,
  "data": {
    "object": {
      "id": "cs_test_123",
      "object": "checkout.session",
      "customer": "cus_test_123",
      "customer_email": "test@example.com",
      "payment_status": "paid",
      "status": "complete",
      "subscription": "sub_test_123"
    }
  },
  "type": "checkout.session.completed"
}
EOF

# Test Stripe webhook (should return 400 without valid signature)
echo "Testing Stripe webhook endpoint..."
response=$(curl -s -w "\n\nHTTP_STATUS:%{http_code}" \
  -X POST https://us-east4-$PROJECT_ID.cloudfunctions.net/stripe-webhook \
  -H "Content-Type: application/json" \
  -H "Stripe-Signature: test_signature" \
  -d @stripe-test-event.json)
  
http_status=$(echo "$response" | grep "HTTP_STATUS:" | cut -d':' -f2)
echo "Webhook test status: $http_status"
if [ "$http_status" = "400" ] || [ "$http_status" = "401" ]; then
  echo " Webhook endpoint is protected (signature validation working)"
else
  echo "  Unexpected status code"
fi

# Test API key validation
echo
echo "Testing API key validation endpoint..."
curl -X POST https://us-east4-$PROJECT_ID.cloudfunctions.net/api-key-validator \
  -H "x-api-key: ctm_test_12345" \
  -H "Content-Type: application/json" \
  -d '{}'

# Clean up
rm stripe-test-event.json

Architecture Decisions

Why Two Repositories?

1. Open Source Integrity: Keep core conversion logic public and MIT licensed
2. Commercial Protection: Billing and subscription code stays private
3. Security: API keys and user data never touch public code
4. Flexibility: Can update pricing without touching public repo

Security Considerations

1. API Key Security

   • Keys stored hashed in Firestore
   • Format: ctm_live_ + 32 random bytes
   • Never logged or exposed

2. CORS Configuration

   • Public endpoints: Allow all origins
   • Commercial endpoints: Restrict to your domains
   • Management endpoints: Strict origin checking

3. Data Privacy

   • Zero-storage architecture maintained
   • User data only in Firestore
   • No file content ever stored
   • Automatic cleanup of processing memory

Cost Optimization

1. Function Configuration

   # Set memory limits based on function type
   --memory=256MB  # API validation, webhooks
   --memory=512MB  # Document conversion
   --memory=1GB    # Large file processing
   

2. Scale-to-Zero Configuration ENABLED BY DEFAULT

   All Google Cloud Functions (Gen2) automatically scale to zero when not in use:

   • Min Instances: 0 (default - enables scale to zero)
   • Max Instances: 100 (configurable via MAX_INSTANCES)
   • Memory: 512MB (configurable via FUNCTION_MEMORY)
   • Timeout: 60 seconds (configurable via FUNCTION_TIMEOUT)

   What this means:

   • Functions automatically shut down when idle (no incoming requests)
   • No costs when not processing requests
   • Cold start latency of ~1-3 seconds for first request after idle period
   • Automatic scaling up to max instances under load

   Current configuration in deployment scripts:

   --max-instances=${MAX_INSTANCES:-100}  # Scale up limit
   # No min-instances specified = 0 (scale to zero)
   --memory=${FUNCTION_MEMORY:-512MB}
   --timeout=${FUNCTION_TIMEOUT:-60}s
   

3. Concurrency Limits

   # Prevent runaway costs
   gcloud functions set-concurrency xlsx-converter --max-instances=10
   

4. Budget Alerts

   # Set up budget alert at $100
   gcloud billing budgets create \
     --billing-account=$BILLING_ACCOUNT_ID \
     --display-name="Convert To Markdown Budget" \
     --budget-amount=100 \
     --threshold-rule=percent=50,basis=current-spend \
     --threshold-rule=percent=90,basis=current-spend
   

Monitoring & Maintenance

Daily Checks

• Function error rates
• API response times
• Stripe webhook success
• Email delivery status

Weekly Tasks

• Review usage patterns
• Check for failed subscriptions
• Monitor Firestore costs
• Update documentation

Monthly Tasks

• Audit API key usage
• Review and optimize costs
• Check for security updates
• Customer success metrics

Troubleshooting Common Issues

Issue: "Permission denied" during project creation

Solution:

# Verify you have the necessary roles
gcloud projects get-iam-policy $(gcloud config get-value project) \
  --flatten="bindings[].members" \
  --filter="bindings.members:user:$(gcloud config get-value account)"

# You need at least:
# - roles/resourcemanager.projectCreator
# - roles/billing.user

Issue: "command not found" in deployment.env

Solution:

# Fix common configuration issues automatically
./fix-deployment-env.sh

# Validate configuration
./validate-config.sh

# Common fixes needed:
# PROJECT_NAME=My Project → PROJECT_NAME="My Project"
# KEY = value → KEY=value (no spaces around =)

Issue: "API not enabled" errors

Solution:

# Enable both required APIs for Gen2 Cloud Functions
gcloud services enable cloudfunctions.googleapis.com \
  run.googleapis.com \
  --project=$PROJECT_ID --quiet

# Or use the script
./03-enable-apis.sh

# Wait longer for propagation
sleep 60

Note: Gen2 Cloud Functions require both cloudfunctions.googleapis.com and run.googleapis.com APIs. If you see a prompt during deployment asking to enable run.googleapis.com, answer 'y'.

Issue: "Function deployment fails"

Solution:

# Check Cloud Build logs
gcloud builds list --limit=5 --project=$PROJECT_ID

# Check function logs
gcloud functions logs read FUNCTION_NAME \
  --limit=50 \
  --project=$PROJECT_ID

Issue: "Firestore already exists in different region"

Solution:

# Firestore can only be created once per project
# Either use existing database or create new project
gcloud firestore databases list --project=$PROJECT_ID

Launch Checklist

Pre-Launch (Development)

• Create private repository
• Set up GCP project
• Configure Stripe products
• Deploy all functions
• Test payment flow
• Configure Gmail API
• Set up monitoring

Launch Day

• Deploy to production
• Update DNS records
• Enable Stripe live mode
• Test live payment
• Monitor first customers
• Announce on social media

Post-Launch

• Monitor error rates
• Gather user feedback
• Optimize performance
• Plan feature updates

Estimated Timeline

Initial Setup: 2-3 hours

• GCP project creation: 15 minutes
• API enablement and verification: 20 minutes
• Firestore setup: 15 minutes
• Stripe configuration: 30 minutes
• Function deployments: 45 minutes
• Testing and validation: 60 minutes

Commercial Features: 1-2 days

• Private repo setup: 2 hours
• Commercial functions: 4 hours
• Email integration: 2 hours
• Webhook configuration: 1 hour
• End-to-end testing: 4 hours

Quick Deployment Script

For experienced users, here's a consolidated deployment script:

#!/bin/bash
# quick-deploy.sh - Consolidated deployment script

set -e

# Configuration
export PROJECT_ID="convert-to-markdown-prod"
export REGION="us-east4"  # Northern Virginia - supports free domain mapping
export BILLING_ACCOUNT_ID="YOUR-BILLING-ID"

# Phase 0: Authentication
gcloud auth login
gcloud config set account YOUR_EMAIL@DOMAIN.COM

# Phase 1: Project Setup
gcloud projects create $PROJECT_ID --name="Convert To Markdown"
gcloud config set project $PROJECT_ID
gcloud beta billing projects link $PROJECT_ID --billing-account=$BILLING_ACCOUNT_ID

# Enable all APIs at once
gcloud services enable \
  cloudfunctions.googleapis.com \
  run.googleapis.com \
  cloudbuild.googleapis.com \
  storage.googleapis.com \
  compute.googleapis.com \
  firestore.googleapis.com \
  gmail.googleapis.com \
  cloudscheduler.googleapis.com \
  logging.googleapis.com \
  monitoring.googleapis.com \
  --project=$PROJECT_ID

# Wait for APIs
sleep 60

# Configure defaults
gcloud config set functions/region $REGION
gcloud config set project $PROJECT_ID

# Create Firestore
gcloud firestore databases create --location=$REGION --project=$PROJECT_ID

echo "Basic setup complete! Continue with function deployment..."

Support Resources

• Google Cloud Functions: Documentation
• Stripe Integration: Stripe Docs
• Gmail API: Gmail API Guide
• Firestore: Firestore Docs

---

Status: This deployment plan covers both open-source and commercial components, providing a complete path from development to production launch.