API Connect Form Factor Migration (FFM)

Throughout 2022-H1 I had the chance to work on a unique API Connect project. The core technical part of the engagement is best described by analogizing it to a brain transplantation surgery. We took the brain out of an existing APIC environment and transplanted into another APIC instance. The source was on-prem v2018 and the target was v10 in AWS.

What is FFM?

The API Connect Form Factor Migration is a technical process to migrate an API Connect instance to another form factor, or to the same form factor. In high level the process consists from the following steps:

1. Set up a new target API Connect instance
2. Extract the APIC data from the source instance
3. Import the extracted data into the target instance
4. Cut over to the target instance

Who could potentially benefit from FFM?

Depending on the existing v2018 (or v10) version and the desired target v10 version, the standard migration path (see documentation below) may require multiple upgrade steps (upgrade version A to B, test, cut over, repeat to version C, repeat to version D, and etc). One of the reasons for someone to go thru FFM is to complete the project in one shot, thus minimizing the risks and significantly reducing the overall project timeline.

Another important reason to use FFM is to change API Connect deployment form factor. The main API Connect supported form factors are VMWare, Kubernetes, Openshift, and CP4I. Many customers start initially with VMWare based deployments. Later, as part of a move-to-containers / move-to-cloud / and other modernization journeys, they could benefit from using FFM to quickly migrate to the new platform.

Lastly, FFM could be used to change the API Connect endpoints. API Connect is a platform. It is based on a micro-services concept, consists from many different services, and these services communicate thru internal and external APIs. Each service is based on endpoints that are set as part of the product installation. Some of these endpoints cannot be changed after deployment. However, there are times that such a change is required. That's where FFM could be helpful as it provides an opportunity to set new endpoints in the target environment. Although it may sound overkill, it is not, and it is the only way (as of now) to change the endpoints.

Reference:

1. API Connect: v2018 to v10 upgrade paths
2. API Connect Upgrade Central: v2018 to v10
3. Migrating v10 to another form factor
4. An API Connect project I used FFM to migrate from v2010 to v10

P.S. The image is from the "Heart of a Dog" film based on a novella by Russian author Mikhail Bulgakov.

IBM API Connect v2018 to v10 migration project

When I was taking project management classes, the instructor mentioned that in real life there are three different Gantt plans - official plan, backup plan, and how the project was executed de facto. He backed this up explaining that many project aspects are unknown at the planning time and for that reason assumptions should be made to minimize risks. He also implied that black belt in assessing scope of work is when the actual number of hours spent on the project is around the number hours initially projected.

As part of my work, I am frequently involved in pre-sales activities. Meeting with clients, learning their challenges, devising solution architecture, and assessing implementation project costs. This is a crucial part since if you go too low you lose financially and if you go too high you might lose the project. I find this part very exciting as it bridges between two different worlds - technology and business.

In 2021 I was working with a client on API Connect Health Check project and later put together a proposal outlining IBM API Connect v2018 to v10 migration scope. Multiple API Connect instances based on hybrid implementation in Openshift and VMWare with multiple gateways and portals. Moving from on-prem to AWS Cloud. Changing the endpoints to support disaster recovery strategy. Migrating all the data and in particular the customer credentials. Now, imagine how challenging it was to put this proposal together. During the last months, I was heads down working on this project.

Long story short, mid of July 2022 the last API Connect environment was successfully cut over to the new v10 instance. It was a long journey with many unknown factors, architecture redesigns, and implementation approach changes, utilizing the Form Factor Migration procedure. Working much more than the usual 8 hours a day, sometimes over weekends, sometimes during the nights. We successfully passed all the different tests including regression, load, performance and the delivery was on time. The client gained a new, stable, modernized, up to date, and best of bread platform to run their core critical business APIs. And for myself, I got to know some great people from the client side, working together to meet the deadlines. Down the road, I think we all deserve a new belt.

Anyone running API Connect make sure you are aware of the end of support (EOS) dates. The API Connect v2018 has announced EOS for April 30, 2023 with no extended support available for purchase.

References

1. Details on the API Connect FFM technique.

API Connect Operator refactoring or how to move a cluster level Operator to a specific namespace

A customer had a single Openshift cluster running three API Connect instances - DEV, UAT, and STAGING. The Operators were installed on a cluster level and centrally managed all the three instances. The customer wanted to upgrade the APIC instances independently at his own pace i.e. upgrade DEV and test it out, then move to UAT, and finally get to STAGING. However, that was not quite possible.

The Operator based API Connect upgrade process consists from upgrading the Operator and then upgrading the Operand. It is a requirement that the Operator and the Operand have the same version and, therefore, implied that the Operand should be upgraded right after the Operator without any major delay. Having a central Operator required the customer to upgrade all three Operands at the same time. We needed to decouple the Operands.

Long story short, after analyzing the underlaying limitations, Openshift architecture, and technology aspects of the possible solutions the decision was made to decentralize the Operator. Instead of having a single, central, cluster level Operator, the proposed to-be architecture was based on separate Operators installed in each Operand namespace. This approach was not a perfect solution as it did not address all of the restrictions. For example, the ibm-common-services and API Connect CRDs remained to be cluster level resources. However, this is a supported topology and clearly allowed the customer to get much closer to their goal with upgrading each API Connect instance separately as long as the cluster level resources do not change or the change was backwards compatible.

The operational part was straight forward - uninstall the central Operators and install them in each Operand namespace. The new Operator auto-discovered the local Operand and took control of the resources. Most pods, including the gateways pods, were restarted. After about an hour the whole instance was healthy. And the best part is that there was no need to reinstall the API Connect instance. Currently, there are three sets of the Operators, one in each Operand namespace. This allows each Operand to be maintained separately from other Operands.

DataPower Refresh from physical to virtual

Introduction

This document provides a high-level overview of the DataPower migration project, also known as the DataPower Refresh project, and deals with migration from physical (hardware) DataPower appliances to virtual (VM, Docker, Linux service) DataPower form factor. This is not meant to be a step-by-step implementation guide but rather a list of topics to discuss and consider as part of a migration planning. The migration should be performed by a certified DataPower SME.

General

1. The process is recommended to be performed as lift and shift technique by migrating a whole appliance at a time rather than breaking it down into groups of services, applications, or application domains.
2. It is generally recommended to perform the migration without making any additional changes (firmware upgrades and etc). If required, work on these changes before the migration begins.
3. As the migration is performed between different DP appliance form factors, Secure Backup method can’t be used, therefore, the migration should utilize the Export/Import Configuration feature.

Review current DP implementation

1. Review architecture, network, HSM, and security implementation.
2. Review service implementation.
3. Check if there are any stateful services in use.
4. Review chained services.
5. Review any non-exportable configuration e.g. IPMI settings, local user accounts, cryptographic files in crypto folders, keytab and LTPA files, keys on HSM. Password Aliases might be missing as well depending on the Domain Settings for password handling.
6. Check if there are any custom files in the store folder.
7. Make sure all crypto files in use are available externally as these aren’t part of the configuration export. If files are missing, plan with the client generation and deployment of new files before the migration.
8. Check firmware version in use vs destination DP version requirements. Should firmware on the source DP be upgraded? Should firmware on the destination DP be downgraded?
9. If firmware upgrade required, check deprecated and removed features in the new version and if there is any impact on the current configuration.
10. Make sure optional modules used by current DP implementation are supported on the destination form factor DP appliance.

Devise migration plan

1. Plan the migration approach -  lift and shift vs coexistence.
2. Aim to avoid downtime (different techniques available for cluster and standalone implementations).
3. Plan how to remove DP from getting new workloads, consider the following scenarios:
• External clients initiate inbound connections to DP – direct or through NLB.
• DP initiates outbound connections for messaging, file transfer, and etc.
• External systems communicate with DP using non-application means.
• Cache and persistent connections.
• Quiesce.
4. Plan if any external to DP configuration should be addressed e.g. new IP addresses, NLB configuration, and etc.
5. Plan for firewall rule changes for inbound requests - API consumers, monitoring, jumpbox, DPOD, and etc.
6. Plan for firewall rule changes for outbound calls - API backends, LDAPs, monitoring,  DPOD, and etc.
7. Plan default domain migration.
8. If required, plan for destination DP appliances cloning.
9. Plan post migration testing strategy:
10. External clients initiate inbound connections to DP – direct or through NLB.
• How to tell if a transaction failed due to the migration?
• How to tell if an object down due to the migration?
11. Decide about the period of post migration support.
12. Review and sign of the migration plan and coordinate tasks with different teams.

Migration process

1. Make sure you have physical access to the old DP appliances.
2. Work according to the devised migration plan.
3. Upload crypto files.
4. Import configuration.
5. Sanity testing.
6. Regression testing.
7. Cut over to the target deployment.

Bibliography 

1. Webcast Replay: IBM DataPower Firmware Upgrades and Migration between Hardware Generations
2. Available modules by product
3. DataPower Gateway for Docker
4. Oracle data sources
5. IBM Support - Migrating a physical DataPower appliance to a virtual edition