Deployment Processes

Service types

Real time services are long living and commonly invoked as web services, although I greatly prefer to have them pull from a queue and scale replicas based on metrics like queue length. Sometimes folks with implement a service as a serverless function which is bad for several reasons including the cold start problem.

Batch jobs (which can be a batch of 1) can run on demand but there’s generally no expectation of real-time execution. Serverless functions are better for this but still not my preferred solution because often batch jobs can be large and long running. Thus batch jobs are often simply a container that executes a script.

Often we need batch jobs that run the same business logic as a real-time service. This is a common pattern in data processing pipelines. In these types of cases, it is best to abstract the business logic into a library that can be used by both batch jobs and real-time services.

Deployment platforms

Deploying to a managed service like ECS or Cloud Run is usually easier to get started but more limiting than something like Kubernetes. However once you have a Kubernetes cluster running it’s easy to deploy new applications (EKS or GKE are fine!). The common reason stated to avoid managed services is to avoid vendor lock-in. This is good idea, but plenty of folks have moved their services from one cloud vendor to the next. The larger problem in my experience is a loss of flexibility. Often managed services limit the number of replicas, the size of the container, or the type of container. They sometimes require a specific framework or library to be used. Perhaps the largest issue is managed services tend to require imperative deployment, which I discuss in the next section.

Declarative vs imperative

Imperative deployments are ones that require the developer define all of the steps to perform to release the application, rather than the desired end state. If any step is missed or fails, the developer is responsible for handling the error handling. This tends to lead to more complex deployments scripts because the developer needs to be aware of all of the steps to perform.

As an analogy, consider a recipe for baking a cake. You might have a recipe for a cake that includes the ingredients, the amount of each ingredient, and the steps to follow to bake the cake. If you follow the recipe, you will end up with a delicious cake. However, if you miss a step or forget to add an ingredient, then you won’t get the desired result.

Declarative deployments consist of specifying the desired end state and letting the tool take care of the rest. In this case, you simply “order the cake” rather than “baking the cake” yourself.

Using tools that support declarative deployments ensures all backing resources are created, if necessary, and can remove unused resources from previous deployments. Tools like Terraform and CloudFormation are great managing infrastructure and ArgoCD is great for managing Kubernetes applications.

Push vs pull

The CI/CD process plays a significant role in whether deployments are performed as a push or pull operation. Most developers are familiar with push style deployments because those are well covered in documentation and examples. These deployments are performed by pushing the deployment at the end of the CI/CD pipeline by running a CLI command or script. This is in contrast to pull style deployments which consist of an operator that watches for changes to a definition of desired state, pulls the desired state, and reconciles the difference between existing and desired states.

A key component of pull style deployments is the ability to update the desired state without having to push a new version of the deployment. This is critical for a production environment because it allows the operator to update the desired state without having to push changes to the code repository or wait through a CI/CD pipeline. Instead you should maintain a deployment repository that contains the desired state for each deployment. This is controversial for some developers because it requires a separate repository, however decoupling the deployment from the code repository enables a more flexible deployment process.

In general, the CI/CD pipeline should push artifacts to a repository. Then the artifact metadata should be used to determine whether to update the desired state. If the artifact should be deployed, the state should be updated, and the operator should reconcile the difference between the desired and actual state.

Recommendations

Based on my experience managing deployments across multiple organizations, here are my key recommendations:

First, prefer declarative deployments over imperative ones. While imperative deployments might seem simpler initially, they become increasingly complex and brittle as your system grows. Tools like Terraform for infrastructure and ArgoCD for applications provide better long-term maintainability.

Second, keep application code and deployment configuration separate. Maintain a dedicated deployment repository that contains your infrastructure as code and application manifests. This separation allows you to:

• Version deployment configurations independently
• Manage access controls separately
• Roll back deployments without changing application code
• Track deployment history clearly

Third, avoid serverless functions unless you have a compelling reason. Instead, consider containerized services that:

• Pull from queues rather than expose HTTP endpoints directly
• Scale based on metrics like queue length or CPU utilization
• Can handle sustained load without cold start penalties

Finally if you’re starting fresh, consider using managed Kubernetes (EKS/GKE) as your deployment platform. While it has a learning curve, it provides:

• A consistent deployment model across different types of applications
• Rich ecosystem of tools and operators
• Flexibility to run both services and batch jobs
• Easy migration between cloud providers if needed

These recommendations focus on creating maintainable, scalable deployment processes that work well for both development and production environments. The key is to optimize for change over time, not just initial deployment ease.

Conclusion

There are many different ways to deploy software and ML models to production. The most important thing is to understand the trade-offs between decisions and choose the right tool for the job. Being able to update your deployments quickly and easily is critical to being able to respond to changes in your business.