Defining multiple portal endpoints for VMWare based API Connect

The Developer Portal stands as a critical subsystem within API Connect, facilitating API Developers' interaction with API Catalogs through an array of technical and collaborative tools. API Developers leverage the Portal for tasks such as API discovery, documentation review, plan subscription, API testing, and issue discussion. To extend access to the Developer Portal for 3rd-party partners, it is common to implement a Web Application Firewall (WAF) that securely exposes the Portal to the internet. This week, a client approached me, expressing interest in exploring the "Defining multiple portal endpoints" feature available on the VMware form factor.

I conducted experimentation on a VMWare-based API Connect stack to evaluate this feature. Initially configured with a single portal-www endpoint, I followed the provided instructions by introducing an extra values file, incorporating three new endpoints. For simplicity, I set all three as aliases to the same portal VM IP address. The system required approximately 5 minutes to reconcile the changes, after which the new endpoints were operational.

To my surprise, I discovered that access to the original Developer Portal Site resulted in an HTTP 404 error, indicating a deviation from its prior functioning state. Evidently, the portal-www endpoints specified in the extra values file superseded the portal-www endpoint in the apiconnect-up-v10.yml file.

Subsequently, I proceeded to configure a new Portal Site. The original behavior, where a single www-portal endpoint was provided based on API Manager selection, persisted. However, the system continued to present the original portal-www endpoint, which was not part of the extra values file. While creating a site with the original endpoint was successful, it remained inaccessible. Conversely, creating a site with a new endpoint proved successful, and the site was accessible.

In summary, this feature represents an integrate capability within API Connect, enabling API developers to access the Developer Portal via distinct endpoints from diverse network segments. Consequently, the same Developer Portal subsystem becomes accessible through endpoints such as ICN (https://developer.int.company.com) and DMZ (https://developer.ext.company.com), catering to different network requirements.

Several considerations must be taken into account when planning the implementation of multiple portal endpoints within an existing API Connect instance. Primary among these considerations is the decision regarding the existing endpoint – whether it should be decommissioned or retained as one of the endpoints. The potential implications of endpoint decommissioning on existing portal users warrant careful evaluation. Additionally, discussions should address the new endpoints' IP addresses, corresponding network segments, and the requisite measures to secure access from untrusted networks, ensuring the overall system's security.

References:

Defining multiple portal endpoints for a VMware environment (API Connect v10.0.5.x)

API Connect Form Factor Migration (FFM)

Throughout 2022-H1 I had the chance to work on a unique API Connect project. The core technical part of the engagement is best described by analogizing it to a brain transplantation surgery. We took the brain out of an existing APIC environment and transplanted into another APIC instance. The source was on-prem v2018 and the target was v10 in AWS.

What is FFM?

The API Connect Form Factor Migration is a technical process to migrate an API Connect instance to another form factor, or to the same form factor. In high level the process consists from the following steps:

1. Set up a new target API Connect instance
2. Extract the APIC data from the source instance
3. Import the extracted data into the target instance
4. Cut over to the target instance

Who could potentially benefit from FFM?

Depending on the existing v2018 (or v10) version and the desired target v10 version, the standard migration path (see documentation below) may require multiple upgrade steps (upgrade version A to B, test, cut over, repeat to version C, repeat to version D, and etc). One of the reasons for someone to go thru FFM is to complete the project in one shot, thus minimizing the risks and significantly reducing the overall project timeline.

Another important reason to use FFM is to change API Connect deployment form factor. The main API Connect supported form factors are VMWare, Kubernetes, Openshift, and CP4I. Many customers start initially with VMWare based deployments. Later, as part of a move-to-containers / move-to-cloud / and other modernization journeys, they could benefit from using FFM to quickly migrate to the new platform.

Lastly, FFM could be used to change the API Connect endpoints. API Connect is a platform. It is based on a micro-services concept, consists from many different services, and these services communicate thru internal and external APIs. Each service is based on endpoints that are set as part of the product installation. Some of these endpoints cannot be changed after deployment. However, there are times that such a change is required. That's where FFM could be helpful as it provides an opportunity to set new endpoints in the target environment. Although it may sound overkill, it is not, and it is the only way (as of now) to change the endpoints.

Reference:

1. API Connect: v2018 to v10 upgrade paths
2. API Connect Upgrade Central: v2018 to v10
3. Migrating v10 to another form factor
4. An API Connect project I used FFM to migrate from v2010 to v10

P.S. The image is from the "Heart of a Dog" film based on a novella by Russian author Mikhail Bulgakov.

IBM API Connect v2018 to v10 migration project

When I was taking project management classes, the instructor mentioned that in real life there are three different Gantt plans - official plan, backup plan, and how the project was executed de facto. He backed this up explaining that many project aspects are unknown at the planning time and for that reason assumptions should be made to minimize risks. He also implied that black belt in assessing scope of work is when the actual number of hours spent on the project is around the number hours initially projected.

As part of my work, I am frequently involved in pre-sales activities. Meeting with clients, learning their challenges, devising solution architecture, and assessing implementation project costs. This is a crucial part since if you go too low you lose financially and if you go too high you might lose the project. I find this part very exciting as it bridges between two different worlds - technology and business.

In 2021 I was working with a client on API Connect Health Check project and later put together a proposal outlining IBM API Connect v2018 to v10 migration scope. Multiple API Connect instances based on hybrid implementation in Openshift and VMWare with multiple gateways and portals. Moving from on-prem to AWS Cloud. Changing the endpoints to support disaster recovery strategy. Migrating all the data and in particular the customer credentials. Now, imagine how challenging it was to put this proposal together. During the last months, I was heads down working on this project.

Long story short, mid of July 2022 the last API Connect environment was successfully cut over to the new v10 instance. It was a long journey with many unknown factors, architecture redesigns, and implementation approach changes, utilizing the Form Factor Migration procedure. Working much more than the usual 8 hours a day, sometimes over weekends, sometimes during the nights. We successfully passed all the different tests including regression, load, performance and the delivery was on time. The client gained a new, stable, modernized, up to date, and best of bread platform to run their core critical business APIs. And for myself, I got to know some great people from the client side, working together to meet the deadlines. Down the road, I think we all deserve a new belt.

Anyone running API Connect make sure you are aware of the end of support (EOS) dates. The API Connect v2018 has announced EOS for April 30, 2023 with no extended support available for purchase.

References

1. Details on the API Connect FFM technique.

API Connect Operator refactoring or how to move a cluster level Operator to a specific namespace

A customer had a single Openshift cluster running three API Connect instances - DEV, UAT, and STAGING. The Operators were installed on a cluster level and centrally managed all the three instances. The customer wanted to upgrade the APIC instances independently at his own pace i.e. upgrade DEV and test it out, then move to UAT, and finally get to STAGING. However, that was not quite possible.

The Operator based API Connect upgrade process consists from upgrading the Operator and then upgrading the Operand. It is a requirement that the Operator and the Operand have the same version and, therefore, implied that the Operand should be upgraded right after the Operator without any major delay. Having a central Operator required the customer to upgrade all three Operands at the same time. We needed to decouple the Operands.

Long story short, after analyzing the underlaying limitations, Openshift architecture, and technology aspects of the possible solutions the decision was made to decentralize the Operator. Instead of having a single, central, cluster level Operator, the proposed to-be architecture was based on separate Operators installed in each Operand namespace. This approach was not a perfect solution as it did not address all of the restrictions. For example, the ibm-common-services and API Connect CRDs remained to be cluster level resources. However, this is a supported topology and clearly allowed the customer to get much closer to their goal with upgrading each API Connect instance separately as long as the cluster level resources do not change or the change was backwards compatible.

The operational part was straight forward - uninstall the central Operators and install them in each Operand namespace. The new Operator auto-discovered the local Operand and took control of the resources. Most pods, including the gateways pods, were restarted. After about an hour the whole instance was healthy. And the best part is that there was no need to reinstall the API Connect instance. Currently, there are three sets of the Operators, one in each Operand namespace. This allows each Operand to be maintained separately from other Operands.

DataPower Refresh from physical to virtual

Introduction

This document provides a high-level overview of the DataPower migration project, also known as the DataPower Refresh project, and deals with migration from physical (hardware) DataPower appliances to virtual (VM, Docker, Linux service) DataPower form factor. This is not meant to be a step-by-step implementation guide but rather a list of topics to discuss and consider as part of a migration planning. The migration should be performed by a certified DataPower SME.

General

1. The process is recommended to be performed as lift and shift technique by migrating a whole appliance at a time rather than breaking it down into groups of services, applications, or application domains.
2. It is generally recommended to perform the migration without making any additional changes (firmware upgrades and etc). If required, work on these changes before the migration begins.
3. As the migration is performed between different DP appliance form factors, Secure Backup method can’t be used, therefore, the migration should utilize the Export/Import Configuration feature.

Review current DP implementation

1. Review architecture, network, HSM, and security implementation.
2. Review service implementation.
3. Check if there are any stateful services in use.
4. Review chained services.
5. Review any non-exportable configuration e.g. IPMI settings, local user accounts, cryptographic files in crypto folders, keytab and LTPA files, keys on HSM. Password Aliases might be missing as well depending on the Domain Settings for password handling.
6. Check if there are any custom files in the store folder.
7. Make sure all crypto files in use are available externally as these aren’t part of the configuration export. If files are missing, plan with the client generation and deployment of new files before the migration.
8. Check firmware version in use vs destination DP version requirements. Should firmware on the source DP be upgraded? Should firmware on the destination DP be downgraded?
9. If firmware upgrade required, check deprecated and removed features in the new version and if there is any impact on the current configuration.
10. Make sure optional modules used by current DP implementation are supported on the destination form factor DP appliance.

Devise migration plan

1. Plan the migration approach -  lift and shift vs coexistence.
2. Aim to avoid downtime (different techniques available for cluster and standalone implementations).
3. Plan how to remove DP from getting new workloads, consider the following scenarios:
• External clients initiate inbound connections to DP – direct or through NLB.
• DP initiates outbound connections for messaging, file transfer, and etc.
• External systems communicate with DP using non-application means.
• Cache and persistent connections.
• Quiesce.
4. Plan if any external to DP configuration should be addressed e.g. new IP addresses, NLB configuration, and etc.
5. Plan for firewall rule changes for inbound requests - API consumers, monitoring, jumpbox, DPOD, and etc.
6. Plan for firewall rule changes for outbound calls - API backends, LDAPs, monitoring,  DPOD, and etc.
7. Plan default domain migration.
8. If required, plan for destination DP appliances cloning.
9. Plan post migration testing strategy:
10. External clients initiate inbound connections to DP – direct or through NLB.
• How to tell if a transaction failed due to the migration?
• How to tell if an object down due to the migration?
11. Decide about the period of post migration support.
12. Review and sign of the migration plan and coordinate tasks with different teams.

Migration process

1. Make sure you have physical access to the old DP appliances.
2. Work according to the devised migration plan.
3. Upload crypto files.
4. Import configuration.
5. Sanity testing.
6. Regression testing.
7. Cut over to the target deployment.

Bibliography 

1. Webcast Replay: IBM DataPower Firmware Upgrades and Migration between Hardware Generations
2. Available modules by product
3. DataPower Gateway for Docker
4. Oracle data sources
5. IBM Support - Migrating a physical DataPower appliance to a virtual edition

One single solution for all OAuth integrations

During the last few months I worked with a few 3rd party vendors to establish new API integrations. These services were JSON based and secured using OAuth i.e. required that the API invocation should carry an Authorization header with previously obtained OAuth access token, obtained against a different OAuth service. Previously, I have described an idea on how to optimize OAuth token management and working on these integrations I had a chance to successfully implement it.

The implementation consisted from a gatewayscript and a configuration file. The gatewayscript core functionality is to manage the OAuth token storage, retrieval, and renewal, and also to set the stage for further business API invocation. The unfortunate issue was that I had to duplicate this implementation for each service as each invocation was slightly different. And so the new challenge was to make this piece of code universal and reusable across the board.

First, I have listed all the places that had service specific hardcoded elements. Next, I designed a solution to externalize those elements as a configuration and changed the code to support external configuration feed. There were approximately 4-5 changes that required rearchitecturing including the ability to dynamically generate headers with static and dynamic values (that's when I recalled what Java reflection is).

Long story short, the upgraded OAuth token management solution allows:

1. OAuth token management to store, retrieve, and renew access tokens;
2. Support multiple OAuth vendors and multiple APIs.
3. Dynamically generates HTTP headers for the business API invocations;
4. Support for static and runtime calculated values for HTTP headers.

The main benefit of this solution is the ease of setting up a new integration that is secured using OAuth. All that is needed - update the configuration file with the expected invocation data and incorporate the OAuth management solution gatewayscript in the processing policy. One single solution for all OAuth integrations.

How to cache an access token in DataPower

Nowadays, SOA architecture is a key basis for any successful integration. Many organizations have opened their gates to allow secure online B2B integration using Rest/SOAP APIs. Millions of transaction are executed each day. API Management solutions make the integration set up part much easier by providing potential partners controlled access to the core API reference. API performance plays crucial role and should be planned already in the design time. Cache is one of the common uses cases that could drastically improve the overall performance. This post will describe how to implement internal DataPower cache to manage and store access tokens.

For this use case let's assume we are integrating with an external partner to invoke a Business API. The API is secured and for each invocation we should provide a Bearer authorization header carrying an OAuth access token. To obtain the access token the partner provides an additional API. An example of a simple, but not necessarily optimized, API design could look like that:

Often, the OAuth access token is provided along with an expiration date, meaning it could be reused multiple times as long as it was not expired. Reusing access token is a great way to reduce network traffic and thus improve the overall runtime latency. In our design the "Obtain Access Token" snippet was adjusted to support access token storage, management, and renewal:

This could be easily achieved by using the DataPower Global Variables as these variables persist beyond the scope of request-response processing. However, keep in mind that use of global variables may introduce unexpected problems as it is not thread safe, there is always a risk of a memory leak, and if used incorrectly can cause DataPower restart.

References

IBM Knowledge Center - DataPower variables

DataPower certification v7.6

DataPower v7.6 certification exam successfully passed at the very first attempt! Brand new addition to my private DataPower certificate collection of 7.1, 5.0, 3.8.1, and 3.7.3 versions!

Running ERTool on MacOS

Updates

Sep 17,2019 - the updated Git package includes full MacOS support. To launch the GUI use the following command:

java -XstartOnFirstThread -jar ErrorReport.jar -gui

The original post

ERTool is a java utility that helps with DataPower error report analysis. I had some issues getting it working on my Mac, so sharing from experience for other people that might benefit from it.

Once downloaded the ERTool project from Git, I've run the build and it failed with compilation errors related to the MacOS environment. This was addressed by changing the build.xml file and MacOS related jar files.

Adding JAR files

Add the following files to the framework/lib folder:

1. org.eclipse.swt-3.1.jar
2. org.eclipse.swt-3.1.jar.zip

Changing the build.xml file

First, add a MacOS environmental property for further references:

<condition property="isMac">     <os family="mac"> </condition>

Next, reference that property while selecting relevant libraries for the build process:

<include if:set="isMac" name="org.eclipse.swt-3.1.jar"/> <include if:set="isMac" name="org.eclipse.swt-3.1.jar.zip"/>

Save the file and rerun the build command. It should complete successfully.

Running the ERTool from command line

Example CLI to run the utility from command line:

java -jar ErrorReport.jar \   -analyticsfile Analytics.xml \   -format HTML \   -outfile output.html \   -file error-report.txt

Bibliography

1. https://www.ibm.com/support/knowledgecenter/en/SS9H2Y_7.7.0/com.ibm.dp.doc/ertool.html
2. https://www.ibm.com/developerworks/community/blogs/de2b7f3e-30c5-4172-be0f-2640ebb55d13/entry/ERTool_Analyze_your_configuration_against_best_practices?lang=en 
3. https://github.com/ibm-datapower/ertool

Enabling DataPower Disaster Recovery Mode without appliance re-initialization

Disaster Recovery Mode is quite an important feature, as it allows simplified migrations from one DataPower appliance to another using the Secure Backup/Restore functionality. This option is set during the initial DataPower configuration. To check if it was enabled, simply access DataPower WebGUI, search for "System Settings" and look up the "Backup mode". If it says "secure", then it was enabled and Secure Backup feature is  available on that appliance.

By the end of 2019 many DataPower gateways would be end of support and clients would face migration projects. Not having Secure Backup / Secure Restore available might cause the projects be more challenging, as it'd require manual configuration copy and dealing with keys.

Until recently, enabling Disaster Recovery Mode required having the gateway re-initialized. The big news are that as of DataPower v2018.4.1.0 firmware it's no longer required. Clients need to open a service ticket to receive the tool to enable secure backup mode.

References:

1. DataPower End Of Support 
2. WebSphere DataPower Release 3.8.1 - Disaster Recovery Mode
3. DataPower v2018.4 - Secure Backup