Continuous Integration/Deployment with Meteor on GitLab CI

In a previous post, I wrote about implementing Continuous Integration/Deployment using Codeship, Docker and Amazon ECS. The project I'm currently working on is hosted on GitLab. GitLab has CI/CD services for free so I set out to take advantage of that.

As usual, there are a several moving parts involved in getting things configured. This article will not cover everything in detail so you should already be familiar with:

How to Test your Meteor Application

How to Deploy Meteor Apps with pm2-meteor

Make sure your work the kinks out of your test and deployment scripts on your local machine before you attempt to configure CI/CD. It can take 10+ minutes to run a job so the debug cycle is time-consuming. It's more efficient to get things working locally where it's easy to make adjustments and re-run a job. Taking a test/deployment process that works on your local machine to GitLab CI is fairly straightforward process of setting up a .gitlab-ci.yml configuration file.

image: blurri/meteor-node

variables:
  METEOR_ALLOW_SUPERUSER: "true"
  METEOR_WATCH_FORCE_POLLING: "true"
  NODE_ENV: 'development'

stages:
  - test
  - deploy

test:
  stage: test
  only:
    - master
    - develop
  script:
    - meteor npm install chimp
    - meteor npm install
    - meteor npm rebuild
    - meteor npm run test-for-ci

deploy:
  stage: deploy
  only:
    - master
    - develop
  when: manual
  script:
    # Install ssh-agent if not already installed, it is required by Docker.
    # (change apt-get to yum if you use a CentOS-based image)
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'

    # Run ssh-agent (inside the build environment)
    - eval $(ssh-agent -s)

    # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
    - ssh-add <(echo "$SSH_PRIVATE_KEY")

    # For Docker builds disable host key checking. Be aware that by adding that
    # you are suspectible to man-in-the-middle attacks.
    # WARNING: Use this only with the Docker executor, if you use it with shell
    # you will overwrite your user's SSH config.
    - mkdir -p ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    # write the ssh key to a file
    - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa

    # install pm2-meteor
    - meteor npm install pm2-meteor -g

    - meteor npm install
    - meteor npm rebuild
    - meteor npm run deploy

Testing

If you are using Chimp to run full acceptance tests, you are probably used to running your Meteor in one terminal and executing the tests in another terminal. When running under a CI pipeline, you'll want to make it so you can execute the test in one script. This discussion has an example of how to do just that. The test script simply installs chimp, installs any packages from package.json, rebuilds and then executes the tests.

Deployment

When deploying with pm2-meteor, you'll need to pass a path to your ssh key so the script can connect to the remote server. This article, provided the information to do that. Once the ssh connection is established, the deploy script can be run.

Good luck setting up your CI/CD pipeline!

Resources:

https://acrofever.com/blog/meteor-1-4-testing-pipeline-gitlab-ci.html

https://ironboundsoftware.com/blog/2016/09/19/deploying-with-gitlab-continuous-integration/

Continuous Deployment of Meteor to Amazon ECS using Docker and Codeship

In the previous article, we looked at what it takes to deploy a meteor app to Amazon EC2 Container Service (ECS). One issue with that deployment process is that the application source code is baked into the docker image. Changes to the application would require creating a new docker image, publishing that image and then stopping running containers and starting new containers pointing to the new image. We'll improve upon that and develop a continuous integration/deployment process using Codeship. Once everything is configured, you will simply commit to your master branch to update your app in production.

 There are a lot of pieces involved. This article will not cover everything in detail so you should be already familiar with:

• using a test framework like Velocity for testing your app
• using Docker with Meteor using MeteorD
• deploying Meteor on Amazon ECS

Make sure you have all those pieces working correctly before you try connecting everything together for continuous delivery. 

 To get everything working you will need to:

• configure Codeship to test your app
• configure ECS to use an app bundle with BUNDLE_URL
• configure Codeship to deploy your app

 Let's get started...

Configure Codeship to test our app

Sign into Codeship and create your project by pointing to your project git repository. Go to Project Settings, then Test Settings. Choose "I want to create my own custom commands". In the Setup Commands enter:

curl -o meteor_install_script.sh https://install.meteor.com/
chmod +x meteor_install_script.sh
sed -i "s/type sudo >\/dev\/null 2>&1/\ false /g" meteor_install_script.sh
./meteor_install_script.sh
export PATH=$PATH:~/.meteor/
meteor --version

 That script basically installs meteor (with a workaround since we're not running with root access on Codeship). Credit to @david_sykora for that script. In the Configure Test Pipelines enter:

meteor run --test

This is a good point to see if the test pipeline is working. Commit to your repo and Codeship will start running the test scripts. If the tests pass you are ready to move on. Note that things run much slower on cloud servers so if your test suite is passing all tests on your local system but not on Codeship that could be the issue. You may need to explicitly wait for DOM elements to appear at certain points in your test scripts.

Configure ECS to use an app bundle with BUNDLE_URL

As mentioned, we don't want to include our app code within the docker container. We'll use the BUNDLE_URL feature of the meteorhacks/meteord image. This allows the image to pull an app bundle from from a url. We'll use S3 to host the app bundle as discussed in this article. Create a bucket in S3 and create a folder for the app bundle. Add a bucket policy to S3 that grants read permissions to all for anything in that folder otherwise the container will fail to start because it can't download the app bundle. Let's get this working manually without Codeship.

On your local machine, create an app bundle by running 

meteor build --architecture=os.linux.x86_64

Upload the tar file to the S3 folder. Refer to the previous article on how to set up Meteor/Docker on ECS. In the task definition, use the BUNDLE_URL variable like this:

{
  "family": "your-app",
  "containerDefinitions": [
    {
      "name": "your-app-container",
      "image": "meteorhacks/meteord:base",
      "cpu": 300,
      "memory": 300,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "environment": [
        {
          "name": "ROOT_URL",
          "value": "http://yourapp.com"
        },
        {
         "name": "BUNDLE_URL",
         "value": "http://yourbucket.s3.amazonaws.com/some-folder/your-app.tar.gz"
        }
      ]
    }
  ]
}

Run the task to verify it is configured correctly. After you confirm the task will run, set up a service to run the task. With the ECS service set up, go back to Codeship and set up deployment.

 Configure Codeship to deploy our app

 We need to configure Codeship with the AWS credentials that can access S3. Go to Project Settings, then Environment and enter variables for: 
AWS_DEFAULT_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

Go to Deployment Settings, under "Add a deployment to your pipeline" there are several options; choose Custom Script. For the custom script enter:

mkdir ~/build
meteor build --architecture=os.linux.x86_64 ~/build/
aws s3 cp ~/build/your-app.tar.gz s3://your-bucket/meteor-bundles/your-app.tar.gz
aws ecs update-service --service web-service --task-definition your-app --desired-count 0
sleep 30
aws ecs update-service --service web-service --task-definition your-app --desired-count 1

The script uses the meteor build command to create a tar file of the application. Then it copies the tar file to S3. It stops the ECS web service by setting the desired-count to 0, waits a bit then starts the service (with a new container that pulls in the latest code).  

Make a commit to your repo and you should see Codeship running through the test and deployment steps; open a browser and find the latest version of your app in production!

Deploying a Meteor App to Amazon EC2 Container Service (ECS)

You've been working on your next killer app and now it's time to deploy to production. There are several ways to deploy a Meteor application; one of those is to use Docker. Once you have figured out how to dockerize your meteor app, the question is where do you deploy your app? Amazon EC2 Container Service (ECS) is an obvious candidate especially if you already use Amazon EC2/AWS.

If you follow along "Deploying a MeteorJS app to ECS", you will be able to deploy a simple meteor app to Amazon ECS. However, if you use the "wizard" and you run into issues or want to use a private docker image it can be challenging to figure things out. I will explain in a bit more detail what's going on so you can deploy your app without using the wizard. We'll use command line tools and configuration files so it will be clear how everything is configured.

Concepts

Between EC2, ECS, Docker, containers, tasks, services, etc. there are a lot of parts and it can get confusing. There's a lot you can do with ECS but let's boil it down for the simple case of serving up a web application.

Cluster - As you would expect, this is a grouping of servers, actually EC2 instances, that serve up your application.

Container Instance - This is just an EC2 instance. Well, not just any instance. It is configured with agent software to talk with the ECS API but it is an EC2 instance.

Task - A task is how you define what the container instance will run. Among other things, this is where you define the docker image (of your app) to run.

Service - A task that should run continuously (versus. run one time) is configured as a service. Web apps typically run as a service.

Prerequisites

You don't have to be an expert with AWS or Docker to follow along but it will be helpful if you are familiar with:

• EC2 instances
• basic docker commands
• AWS command line tools
• EC2 key pair
• SSH - to access your instances

If you don't have aws-cli installed, go do that and come back. 

To get the app up and running we will need to:

• Create an IAM role
• Configure and launch an EC2 instance
• Configure the task
• Run the task (as a service)

Create an IAM role

As mentioned above, the container instances have agents that need to communicate with the ECS API. We must create an IAM role for the container instances to use when they are launched. If you ran through the getting started wizard in ECS you probably already have the required "ecsInstanceRole". If you don't have this role, create it by following these instructions.

Configure and launch an EC2 instance

Create a .json file like this: 

{
  "DryRun": false,
  "ImageId": "ami-a39df1c3",
  "KeyName": "my-amazon-key-pair",
  "SecurityGroups": [
    "my-security-group"
  ],
  "InstanceType": "t2.micro",
  "IamInstanceProfile": {
    "Name": "ecsInstanceRole"
  }
}

• "ImageId" - Use an ECS-optimized image for your region. They are listed here. I'm in region "us-west-1" so I use "ami-a39df1c3". 
• "KeyName" - the name of your key-pair you will use to SSH into your instance.
• "SecurityGroups" - name of the security group.
• "InstanceType" - whatever instance type you like. For testing, I use "t2.micro".
• "IamInstanceProfile" - refer to the ecsInstanceRole we created.

Now launch the instance with:
    aws ec2 run-instances --cli-input-json file://ec2-instance.local.json

 You can check the EC2 console to see the status of the instance. Try connecting to the instance via SSH. 

Configure the task

Create a .json file like:

{
  "family": "my-task",
  "containerDefinitions": [
    {
      "name": "my-container",
      "image": "dockeruser/image",
      "cpu": 300,
      "memory": 300,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "environment": [
        {
          "name": "ROOT_URL",
          "value": "http://mydomain.com"
        }
      ]
    }
  ]
}

• family - basically the name of your task 
• containerDefinitions
• name - name the container whatever you like
• image - the docker image used to start your container
• cpu - 300 is fine for testing
• memory - 300 is fine for testing
• environment - the environment variables to pass to the container

Register, i.e. create, the task by running:
    aws ecs register-task-definition --cli-input-json file://my-task-definition.json

Run the Task

Once the instance is running and the task is defined, simply run the task with:
    aws ecs run-task --cli-input-json file://run-task.json

If all goes well, you should be able to browse your application on port 80 of the public IP of your container instance. You can go into the ECS control panel, drill into the default cluster and view the running task.

When you are done with the task, click "Stop" in the control panel or use the cli:

    aws ecs stop-task --task-id (the-task-id)

 It is important to note that although the task has stopped the EC2 instance is still running. You will be billed for EC2 usage for the running instance even if you are not running a task. 

 If you want your site to run continuously, run it as a service. To create the service:
    aws ecs create-service --service-name my-web-service --task-definition my-task --desired-count 1

 If you want to stop the service you'll notice the ECS control panel doesn't have a stop button for the service. There is a delete button but if you click it you get an error: 

 "Unable to delete service -The service cannot be stopped while the primary deployment is scaled above 0."

 So how do you stop this thing?  You can update the existing service and set the desired-count to 0

    aws ecs update-service --service my-web-service --desired-count 0

 That stops the task running under the service and now you can delete the service if you want.

 Have fun deploying! In my next blog post, I'll show you how to connect to a private docker repository.

Resources

https://github.com/philcruz/docker-ecs
AWS CLI Reference
ECS Command Line Reference

Manually Getting OAuth Tokens in Meteor

Meteor is a great framework that provides several features/packages that accelerate application development. One very useful feature is Meteor Accounts. With just a few lines of code you can add a full featured user account system that supports login, logout, account creation and support for OAuth providers like Google and Facebook. If you just want to get things up and running there's no need to understand the details of the OAuth mechanism as Meteor handles that for you automagically. However, there may be times when you don't want to or can't use the Meteor Accounts package to do OAuth. Perhaps, the application needs to access an account that is different from the logged in user, like a shared support email account. In this case, you need to handle the OAuth manually. Fortunately, handling OAuth with Meteor is fairly straightforward. Let's look at what that entails.

We'll look at the process for Google OAuth but the same concept should apply to other providers. The basic steps are:

• Create client credentials (client_id, client_secret) in the control panel for the OAuth provider. This will including setting up the redirect_uri 
• Generate the login Url using the client_id, client_secret, and redirect_uri
• Implement the redirect_uri in your app to capture the auth code 
• Exchange the auth code for an access token (and refresh token) 
•  If the access token expires, get a new access token using the refresh token

Create Client Credentials

In the Google Developer Console, go to the project then APIs & auth > Credentials. Click to "Create new Client ID" and you will see something like:

For the authorized JavaScript origins and Authorized redirect URIs you can enter: http://localhost:3000 and http://localhost:3000/oauth.

Generate the login URL

In your login template, create a helper to generate the login URL using the client credentials. For example:

Template.login.helpers({

    loginUrl: function() { 
        var loginUrl = "https://accounts.google.com/o/oauth2/auth?scope=" + Meteor.settings.public.scope
            + "&redirect_uri=" + Meteor.settings.public.redirect_uri
            + "&response_type=code&client_id=" + Meteor.settings.public.client_id
            + "&access_type=offline";
        return loginUrl;
    }
});

Implement the redirect_uri

If the redirect_uri is set to http://localhost:3000/oauth, then set up a route to this url. Using iron:router:

Router.route('/oauth',  {name: 'oauth'} );

In the "oauth" template, we'll set up a button to get the authentication code from the url and call a Meteor method to exchange the code for an access token. In oauth.js we'll implement a function to handle the click:

Template.oauth.events({
    'click #getTokensButton': function(){        
        var controller = Iron.controller();
        var code = controller.getParams().query.code;
        Meteor.call("exchangeCodeForTokens", code, function(error, results) {
            console.log(results); 
        });
    }
});

Here is the Meteor method:

Meteor.methods({
    
    exchangeCodeForTokens: function (code) {
        this.unblock();
        var apiUrl = "https://www.googleapis.com/oauth2/v3/token";
        try 
        {        
            var result = HTTP.post( apiUrl,
                {
                params: {                                        
                    'code': code,
                    'client_id': Meteor.settings.public.client_id,
                    'client_secret': Meteor.settings.client_secret,
                    'redirect_uri': Meteor.settings.public.redirect_uri,
                    'grant_type': 'authorization_code'
                }
            });            
            Tokens.upsert(
                {account: Meteor.settings.account},
                { $set:
                    {                        
                        accessToken: result.data.access_token,
                        expiresIn: result.data.expires_in
                        
                    }
                }
            );
            if (result.data.refresh_token) {
                Tokens.upsert(
                    {account: Meteor.settings.account},
                    { $set:
                        {                        
                            refreshToken: result.data.refresh_token
                        }
                    }
                );                
            }
            return result;         
        } catch(error){
            console.log(error)
            return error;
        }        
    }
});

Make a call to the Google OAuth API with the authorization code. The response will contain the access_token which you can save in a collection. In the login URL above, we specified "access_type=offline" so the response will also include a refresh token. Access tokens expire after a certain amount of time like around 1 hour. Once the token expires you will either have to login in again to get a new authorization code and exchange it for a new token or if you obtained a refresh token you can exchange the refresh token for a new access token. The Meteor method to exchange the refresh token looks like:

Meteor.methods({
    exchangeRefreshToken: function(){
        console.log("in exchangeRefreshToken...");        
        this.unblock();            
        var tokens = Tokens.findOne();            
        console.log(tokens.refreshToken);
        var apiUrl = "https://www.googleapis.com/oauth2/v3/token";
        var result = HTTP.post( apiUrl, {
            params: {
                'client_id': Meteor.settings.public.client_id,
                'client_secret': Meteor.settings.client_secret,
                'refresh_token': tokens.refreshToken,
                'grant_type': 'refresh_token'
            }
        });
        
        if (result.data.access_token){
            Tokens.upsert(
                {account: Meteor.settings.account},
                    { $set:
                        {                        
                            accessToken: result.data.access_token,
                            expiresIn: result.data.expires_in
                        }
                    }                
            );            
        }
        return result.data;
    }

});

That's all there is to it! If  you want to see an example app that implements this, check out https://github.com/philcruz/meteor-gmail-example.

More information:
Google - Using OAuth 2.0 for Web Server Applications

Stripe and ColdFusion

If you are trying to integrate credit card processing into your website and you're wondering why it is has to be so much trouble, check out Stripe. Stripe is a simple, developer-friendly way to accept payments online. If you want to integrate Stripe with your ColdFusion application, check out Stripe.cfc.

This blog has moved

This blog is now located at http://blog.philcruz.com/.
You will be automatically redirected in 30 seconds, or you may click here.

For feed subscribers, please update your feed subscriptions to
http://blog.philcruz.com/feeds/posts/default.

BlueDragon goes Open Source

Yesterday, Vince from New Atlanta announced on his blog that they are releasing BlueDragon/J2EE as open source. You can read more details from the faq but the gist of it is that it's a fully functional CFML engine that you can use for commercial situations for free (with limitations as far as distribution). This is great news for the CF community. Maybe this will breathe new life into CFEverywhere type applications.