blog

How to synchronise AWS ECS Services

​​Author: Stoyan Gramatikov, AWS Cloud Infrastructure Engineer

In this blog post, I’ll outline how to implement a custom AWS ECS solution. One that allows you to overcome a common challenge often experienced by businesses on AWS: not having a built-in synchronisation mechanism that can implement dependencies between AWS ECS services.

Amazon Elastic Container Service (ECS) is a highly scalable, high-performance container orchestration service that supports Docker containers and allows you to easily run and scale containerized applications on AWS.

As an AWS Advanced Consulting Partner, HeleCloud works with its customers to overcome traditional business barriers through the adoption of cloud services and solutions. Recently, we helped a large electronic financial markets firm to deliver the key components of its secure and robust trade automation platform.

To do this, we [HeleCloud architecture team] selected AWS ECS as it allowed us to manage containers and allowed developers to run applications in the Cloud without having to configure an environment for the code to run it. Despite using a microservices architecture, which was implemented using Docker containers and run on Kubernetes, there remained a challenge over the synchronisation of dependencies.

In (and out of) sync

Most of the applications developed for the customer, required interactions between different services that were dependent on each other. These dependencies required starting services in a well-known order, specific for each application. Whilst AWS ECS had instruments to synchronise dependencies between containers within a service, using the keyword “dependsOn” within task definitions, total synchronisation between different services remained a serious challenge.

In order to create a successful synchronisation solution, the answer must center around modifying the dependent image. This can be achieved by plugging in a piece of script code that is executed before any other application code in the container. Once the dependency is fulfilled, it makes the contained executing its application code.

The solution eliminates the need for you to install and operate your own container orchestration software, manage and scale a cluster of virtual machines, or schedules containers on those virtual machines as if it was created from the original image.

​​AWS ECS Basics Architecture

Two bash scripts are doing the job:

  1. sync-waiter-make-config.sh – bash script creating the image specific plug-in script and creating a new modified image;
  2. sync-waiter-this_YYYY-MM-DD_hh:mm:ss.nanosecs.sh – plug-in bash code, autogenerated by sync-waiter-make-config.sh.

The plug-in has two parameters, provided as environment variables:

  1. Variable name: CHECK_URLs – expects a list of URLs separated by spaces;
  2. Variable name: CHECK_INTERVAL_SECONDS – number of seconds between retries to access an URL.

The algorithm to create a new image with the plugged-in bash code:

  1. If your docker registry hosting the original image requires authentication, make sure you logged in to it before going to the next step;
  2. Run sync-waiter-make-config.sh by passing one parameter with the original image URL.
    For example:
    sync-waiter-make-config.sh quay.io/alfresco/alfresco-digital-workspace:1.2.0
  3. The script installs “curl” tool. This part of the script might need to be modified, depending on the type of OS in your original image;
  4. The script reads the ENTRYPOINT and CMD of the original image and saves them;
  5. The script generates a new image with name as the original one with capital “S” appended at the end. For example:
    quay.io/alfresco/alfresco-digital-workspace:1.2.OS
  6. The new image ENTRYPOINT and CMD are modified so that firstly plug-in is executed and after the dependencies are fulfilled the original application code is executed.

The algorithm for executing the modified container:

  1. Bash plug-in is executed first on starting the container from a modified image;
  2. Plug-in reads the environment variables values of CHECK_URLs and CHECK_INTERVAL_SECONDS;
  3. The plug-in checks the connectivity to the first URL;
  4. If there is successful check of the URL plug-in goes to check the next one;
  5. If check fails plug-in waits CHECK_INTERVAL_SECONDS seconds and retries;
  6. After all the URLs are successfully checked the plug-in executes ENTRYPOINT and CMD code of the original image.

When implementing this solution, it is important to remember that it is focused only on direct network connectivity dependencies. For any other types of dependencies, such as shared storage, bash scripts must be updated. Secondly, the solution hasn’t been tested on all the docker image types. It exposes only the idea of overcoming the given problem. For some images, you may need to modify the part of Bash scripts creating the “Dockerfile”, especially in “curl” installation piece.

Whilst Bash scripting may seem daunting, it is an extremely useful and powerful part of system administration and development. Utilizing its capabilities in this way can help achieve synchronization between AWS ECS services.

The Bash Script

#!/usr/bin/env bash
#
# Developed by Stoyan Gramatikov @ Helecloud LTD., Sofia, Bulgaria
#

# Verify function parameters
if [ ${#} -ne 3 ] || ! echo “${1}” | grep -q ‘:\S’ || [ “${1}x” == “x” ] || [ “${2}x” == “x” ] || [ “${3}x” == “x” ]
then
{
echo
echo ‘Usage  : ‘”${0}”‘ <image path> <DEFAULT_CHECK_URLs> <DEFAULT_CHECK_INTERVAL_SECONDS>’
echo ‘Example: ‘”${0}”‘ “docker.io/alfresco/alfresco-content-repository-aws:6.1.1.1” “https://www.yahoo.com https://www.google.com” “30”‘
echo
} 1>&2

exit 128
fi

#################################### BEGIN ###################################
########################## PACKAGE this_sync_waiter ##########################

# Set default parameters
function this_sync_waiter_set_default_parameters()
{
if [ ${#} -ne 2 ]
then
{
echo
echo ‘Usage   : ‘”${FUNCNAME[0]}”‘ <default CHECK_URLs> <default CHECK_INTERVAL_SECONDS>’
echo ‘Example : ‘”${FUNCNAME[0]}”‘ “https://www.yahoo.com https://www.google.com” “30”‘
echo
} 1>&2

return 128
fi

local default_check_urls=”${1}”
local default_check_interval_seconds=”${2}”

if [ “${CHECK_URLs}x” == “x” ]
then
CHECK_URLs=”${default_check_urls}”
fi

if [ “${CHECK_INTERVAL_SECONDS}x” == “x” ]
then
CHECK_INTERVAL_SECONDS=”${default_check_interval_seconds}”
fi
}

# Print parameters
function this_sync_waiter_print_parameters()
{
echo “CHECK_URLs=${CHECK_URLs}”
echo “CHECK_INTERVAL_SECONDS=${CHECK_INTERVAL_SECONDS}”
}

# this_sync_waiter_export function
function this_sync_waiter_export()
{
declare -f this_sync_waiter_set_default_parameters
declare -f this_sync_waiter_print_parameters
declare -f this_sync_waiter_main
}

# this_sync_waiter_main function
function this_sync_waiter_main()
{
if [ ${#} -ne 2 ]
then
{
echo
echo ‘Usage   : ‘”${FUNCNAME[0]}”‘ <default CHECK_URLs> <default CHECK_INTERVAL_SECONDS>’
echo ‘Example : ‘”${FUNCNAME[0]}”‘ “https://www.yahoo.com https://www.google.com” “30”‘
echo
} 1>&2

return 128
fi

# Set default parameters if no values are set.
this_sync_waiter_set_default_parameters “${@}”

echo ‘### Parameters used …’
this_sync_waiter_print_parameters

echo ‘### Start checking …’

local urls=”$( echo “${CHECK_URLs}” )”
local url=
local i=

for url in ${urls}
do
echo ; echo “### Checking URL: ${url}”

i=0
while true
do
let “i++”
echo “— Check number: ${i}”
echo “— Executing: curl -Ls ${url} > /dev/null”
if curl -Ls “${url}” > /dev/null
then
echo “— Successfully checked.”
break
else
echo “— Waiting ${CHECK_INTERVAL_SECONDS} seconds …”
sleep “${CHECK_INTERVAL_SECONDS}”
fi
done
done

echo “### Continuing …”
}

########################## PACKAGE this_sync_waiter ##########################
##################################### END ####################################

SRC_IMAGE=”${1}”
DEFAULT_CHECK_URLs=”${2}”
DEFAULT_CHECK_INTERVAL_SECONDS=”${3}”

DST_IMAGE=”${SRC_IMAGE}S”
SYNC_MARKER=”$( date +%Y-%m-%d_%H:%M:%S.%N )”
SYNC_WAITER_THIS_SCRIPT=”sync-waiter-this_${SYNC_MARKER}.sh”
DOCKERFILE_NAME=”Dockerfile-this_${SYNC_MARKER}”

# Create this sync waiter script
function sync_waiter_create_this_script()
{
echo -n “~~~ Creating ${SYNC_WAITER_THIS_SCRIPT} … ”

{
this_sync_waiter_export

echo ; echo ‘# pre sync waiter main routine’
echo “:”

echo ; echo ‘# sync waiter main routine’
echo “this_sync_waiter_main ‘${DEFAULT_CHECK_URLs}’ ‘${DEFAULT_CHECK_INTERVAL_SECONDS}'”

echo ; echo ‘# Add-on’

local image_info=”$( docker image inspect “${SRC_IMAGE}” )”
local rc=${?}

if [ ${rc} -ne 0 ] || [ “${image_info}x” == “x” ]
then
echo ‘ERROR: Failed to get source image info: ‘”${SRC_IMAGE}”  1>&2
return 2
fi

local entrypoint=$( echo “${image_info}” | jq ‘.[].Config.Entrypoint’ )
local cmd=$( echo “${image_info}” | jq ‘.[].Config.Cmd’ )

if echo ${entrypoint} | grep -q ‘^\s*\[‘
then
entrypoint=$( echo “${entrypoint}” | jq ‘.[]’ )” ”
else
entrypoint=”
fi

if echo ${cmd} | grep -q ‘^\s*\[‘
then
cmd=$( echo “${cmd}” | jq ‘.[]’ )
else
cmd=”
fi

echo ${entrypoint}${cmd}

}  > “${SYNC_WAITER_THIS_SCRIPT}”

rc=${?}

if [ ${rc} -eq 0 ]
then
echo ‘Done.’
fi

return ${rc}
}

# Create Dockerfile
function sync_waiter_create_dockerfile()
{
echo -n “~~~ Creating ${DOCKERFILE_NAME} … ”

{
echo “FROM ${SRC_IMAGE}”

echo ‘USER root’
echo ‘RUN ( apk add –update curl && rm -rf /var/cache/apk/* ) || ( yum install curl -y ) || ( apt-get update -y && apt-get install curl -y )’

echo “COPY ${SYNC_WAITER_THIS_SCRIPT} /.”
echo “ENTRYPOINT [ \”/bin/sh\” ]”
echo “CMD [ \”/${SYNC_WAITER_THIS_SCRIPT}\” ]”

} > “${DOCKERFILE_NAME}”

rc=${?}

if [ ${rc} -eq 0 ]
then
echo ‘Done.’
fi

return ${rc}
}

# sync_waiter_main routine
function sync_waiter_main()
{
sync_waiter_create_dockerfile && sync_waiter_create_this_script && docker build -f “${DOCKERFILE_NAME}” -t “${DST_IMAGE}” .
rc=${?}

if [ ${rc} -eq 0 ]
then
echo
echo “Synced image produced: ${DST_IMAGE}”
echo
fi

return ${rc}
}

# Execute sync_waiter_main routine
sync_waiter_main

Any questions, please feel free to get in touch through my LinkedIn or take a look at how HeleCloud is helping businesses embrace Cloud technologies.

References

AWS ECS guide:
https://docs.aws.amazon.com/AmazonECS/latest/userguide/Welcome.html
Docker file reference:
https://docs.docker.com/engine/reference/builder/
Bash reference manual:
https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html