Non-IBM Disclaimer

The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.

Thursday, January 31, 2019

Running ERTool on MacOS

Updates

Sep 17,2019 - the updated Git package includes full MacOS support. To launch the GUI use the following command:
java -XstartOnFirstThread -jar ErrorReport.jar -gui

The original post

ERTool is a java utility that helps with DataPower error report analysis. I had some issues getting it working on my Mac, so sharing from experience for other people that might benefit from it.

Once downloaded the ERTool project from Git, I've run the build and it failed with compilation errors related to the MacOS environment. This was addressed by changing the build.xml file and MacOS related jar files.

Adding JAR files

Add the following files to the framework/lib folder:
  1. org.eclipse.swt-3.1.jar
  2. org.eclipse.swt-3.1.jar.zip

Changing the build.xml file

First, add a MacOS environmental property for further references:
<condition property="isMac">
    <os family="mac">
</condition>

Next, reference that property while selecting relevant libraries for the build process:
<include if:set="isMac" name="org.eclipse.swt-3.1.jar"/>
<include if:set="isMac" name="org.eclipse.swt-3.1.jar.zip"/>

Save the file and rerun the build command. It should complete successfully.

Running the ERTool from command line

Example CLI to run the utility from command line:
java -jar ErrorReport.jar \
  -analyticsfile Analytics.xml \
  -format HTML \
  -outfile output.html \
  -file error-report.txt

Bibliography

  1. https://www.ibm.com/support/knowledgecenter/en/SS9H2Y_7.7.0/com.ibm.dp.doc/ertool.html
  2. https://www.ibm.com/developerworks/community/blogs/de2b7f3e-30c5-4172-be0f-2640ebb55d13/entry/ERTool_Analyze_your_configuration_against_best_practices?lang=en 
  3. https://github.com/ibm-datapower/ertool

Tuesday, January 1, 2019

Enabling DataPower Disaster Recovery Mode without appliance re-initialization

Disaster Recovery Mode is quite an important feature, as it allows simplified migrations from one DataPower appliance to another using the Secure Backup/Restore functionality. This option is set during the initial DataPower configuration. To check if it was enabled, simply access DataPower WebGUI, search for "System Settings" and look up the "Backup mode". If it says "secure", then it was enabled and Secure Backup feature is  available on that appliance.

By the end of 2019 many DataPower gateways would be end of support and clients would face migration projects. Not having Secure Backup / Secure Restore available might cause the projects be more challenging, as it'd require manual configuration copy and dealing with keys.

Until recently, enabling Disaster Recovery Mode required having the gateway re-initialized. The big news are that as of DataPower v2018.4.1.0 firmware it's no longer required. Clients need to open a service ticket to receive the tool to enable secure backup mode.

References:
  1. DataPower End Of Support 
  2. WebSphere DataPower Release 3.8.1 - Disaster Recovery Mode
  3. DataPower v2018.4 - Secure Backup

Sunday, August 19, 2018

DataPower High Availability support overview

Introduction
Nowadays, many organizations can no longer exist in a "silo" mode. To keep up with extremely fast changing world they are required to allow bidirectional integration with clients and partners for both consuming 3rd party APIs and exposing their-own. Many of them choose IBM DataPower Gateways to keep these integrations secured. Any outage in the SOA middleware infrastructure might cause companies to lose business, clients and credibility. To keep the ball rolling most companies implement High Availability across in the stuck to eliminate single point of failure by adding redundancy to the system, so that failure of any single component does not compromise the entire system. This post describes High Availability patterns supported by the DataPower appliances for inbound transactions.

HA option 1 - Standby Control
DataPower Standby Control feature is based on VRRP technology,  allows operation of two or more DataPower appliances in Active/Standby mode. Standby Control provides HA between two different network interfaces; is part of the NIC configuration; and allows interface failover to the standby NIC (usually located on another appliance). NIC with the highest priority is the active NIC, it owns the  the VIP, receives all inbound requests, processes them and returns results. All other NICs in the HA group are in a standby mode; configured with lower priority; waiting for the active NIC to fail to take over the VIP.
This feature is included with all DataPower appliances and doesn't require purchasing any additional licenses. In case of virtual appliances, docker appliances, and appliances running in cloud, in some cases it is possible to setup Standby Control, however, there are some restrictions that should be addressed prior proceeding with this option. It is possible to have this HA pattern configured on two different NICs on the same appliance, in older firmware versions, however, it wasn't supported.

HA option 2 - Self Balancing
Standby Control is nice, but it doesn't scale. And that's exactly the benefit of Self balancing pattern. Self balancing is provided by DataPower Option for Application Optimization. AO is a field upgradable option that requires an additional license i.e. has it's own part number. Self balancing is based on the Standby Control feature and extends its capabilities to support active/active mode. One NIC is the Distributor, it owns the VIP, receives all inbound requests and distributes them for further processing among all other NICs (appliances) in a group, including itself.

HA option 3 - External Network Load Balancer
In this architecture HA is configured, deployed and operates on external NLB. The configuration and features are dependent on the NLB vendor and therefore won't be described as part of this technote.

Summary
What would be a good fit for your organization? It depends on many different aspects. The Standby Control is a great option and a good starting point especially with new DataPower implementations. If you require active/active operation, using external NLB might seem as a very straightforward approach, especially in organizations with adopted NLB solution. Indeed, it has many advantages:
  1. It allows active/active mode without purchasing additional DataPower license for AO.
  2. Existing and skilled Load Balancer team is there to assist with any issue.
  3. Procedures and runbooks are written and in use.
  4. Having NLB implemented as a reverse-proxy, would add another level of security.
  5. In addition, you'd prefer to keep all load balancing practice in one place, wouldn't you?
However, in the long run this approach might introduce some challenges. For example:

  1. Working in Active-Active topology might cause the troubleshooting process last longer and be more complicated, as now there are two or more DataPower Gateways that each of them could potentially run the transaction in question. This risk can be managed by using DataPower Operations Dashboard.
  2.  The inbound transactions source IP would now be the external NLB IP. Will that impact your troubleshooting techniques and authentication policies? Would it be possible to disable the Source IP NAT on the NLB?
  3. Depending on the NLB vendor, healthcheck requirements and the chosen implementation, each DataPower FSH might be required to be represented as a separate farm on NLB, thus making  DataPower development processes longer, dependent on other external resources, and, therefore, more complicated. Having in place a defined policy for FSH use would simplify this risk.
  4. Having network SSL/TLS encrypted, in some cases NLB should have the ability to inspect data (headers and payload), which might involve additional licensing costs for purchasing NLB SSL terminator component and impact on overall performance.
  5. If  SSL caching is implemented, adding NLB might cause problems.
  6. Adding another point of failure might make the troubleshooting process more complicated and longer.
What would be a best fit for your organization? Feel free to reach out with any questions. 

Bibliography
  1. Understanding Standby Control and Load Balancing for WebSphere DataPower SOA Appliances.
  2. Virtual Router Redundancy Protocol.
  3. DataPower option for Application Optimization.
  4. Standby control and self balancing enhancements as of firmware v6.

Thursday, May 3, 2018

DataPower as SFTP Gateway

DataPower is not only about HTTP traffic. In quite a few implementations, DataPower is being used as a central SFTP gateway, providing secured interface, allowing external clients and partners to transfer files. In this post I'll try to briefly point out some of the main advantages using DP for transferring files.

Some high level security aspects of SFTP solution on DataPower:
  1. SFTP is Secure File Transfer Protocol piggy-backing SSH protocol, so the connection between client and server nodes is encrypted to protect against attacks.
  2. DP acts as a reverse proxy breaking the incoming connection and thus preventing direct access to the destination server / internal network.
  3. DP AAA feature can be used to authenticate client's credentials and authorize the operation.
  4. ACL policies can be used to enforce restrictions on client IP addresses accessing the service.
  5. Support isolated tenants is possible by utilizing Content Based Routing feature to implement dynamic routing, so destination folder (and/or server) would be determined and set separately for each transaction during the runtime according to the request data. For example if I am logging in with "GOSHA" userid, the destination might be set to ftp://internalfss.com/external/GOSHA/.
  6. Files can be sent for inspection to one of the supported ICAP-enabled antiviruses.
Some tips with regards to the implementation:
  1. Usually you would use Multi Protocol Gateway with SFTP Front Side Handler. The backend could be regular (S)FTP server. If the backend is based on IIS, you might want to have "Directory Listing Style" set to "UNIX", otherwise connected clients might have issues with listing files.
  2. If not set already, it would be a good practice to have dedicated ETH interfaces for each network, i.e. "INTERNAL" interface and "EXTERNAL" interface with corresponding Host Aliases.
  3. The AAA should be configured on the SFTP Front Side Handler level, not as part of the processing policy.
  4. Make sure you allow empty messages processing.
  5. Don't forget to adjust maximum message size, since the default is only 4 MB.
This is not a step-by-step guide and many configuration steps weren't outlined since a lot depends on the security policy and business requirements. By implementing this integration pattern, organizations gain the ability to quickly on-board new SFTP clients, bring new business to the company in a fast and agile way, utilize already existing central file transfer infrastructure.

Fell free to reach out if you have any questions.

Friday, February 9, 2018

DataPower migration

The following technote outlines lifecycle dates for DataPower hardware appliances. In a nutshell, currently there are numerous DataPower hardware appliances that are supported under IBM S&S contract:
  • XG45, XI52 and XB62 will be end of life on Dec 31, 2019
  • IDG has no end-of-life date published yet. However, since the GA was on Nov 2014, the EOS date might be published soon. (UPD - IDG end-of-service and end-of-life dates were published - June 30, 2023).
  • New IDG X2 has been recently announced and generally available as of March 2018 or June 2018, depending on if HSM module is required or not.

IBM support beyond the end-of-life date might be possible, however, customers would need to purchase an extended support option, which is not always available and might not be cost effective. An alternative would be migrating to supported hardware or virtual models. Migration should be carefully planned and coordinated with various internal teams (network, business, security etc), external partners and customers. Overall, the migration process consists from the following stages:
  • initial installation - rack mounting, network, firmware upgrades.
  • moving all configuration from old appliances - probably using Secure Backup & Secure Restore capability. Otherwise, if migrating manually, it might require significant effort especially with the crypto files i.e. certificates and private keys
  • making changes due to the destination appliance characteristics. For example, the ETH naming convention in IDG appliances was changed comparing to previous 9235 type and migrating from older appliances required manual adjustments.
  • testing - make sure the migration didn't cause any new issues.
  • official go live.

Note, as for DataPower XG45 and XI52 virtual appliances - it is possible to convert old models into IDG using IBM provided tool.

Some questions to consider while planning the migration:
  1. Is it possible to use the secure backup option for the migration?
  2. If going the export configuration path, are all the crypto files available externally to the old DataPower appliance?
  3. Do you prefer the same day cut over migration or coexistence? Each option has different impacts and requirements from the IT perspective.
Are you ready for this End Of Support? Feel free to reach out with any question.

Useful links:
  1. Secure Backup/Restore considerations
  2. Disabling a feature/module  
  3. Migrating a physical DataPower appliance to a virtual edition

Thursday, January 11, 2018

Failure upgrading DataPower to v7.6

Updates

Oct 12,2019
https://www.ibm.com/support/pages/known-limitations-affecting-datapower-virtual-appliance-different-environments

Original post

I was upgrading DataPower firmware. The upgrade procedure was in place and followed many times. The correct firmware version was downloaded from IBM fix central. The unsupported and deprecated features were analyzed for possible impact on existing implementation. The maintenance windows was scheduled. Everything was ready for the upgrade.

When a time has come, the firmware was uploaded to the appliance, the upgrade kicked-off and the appliance rebooted. The execution was perfect and everything was just according to the plan. Well, not everything - the upgrade failed.

DataPower Virtual Edition users planning to upgrade firmware to v7.6.x should check whether the new hardware requirements change has any impact on them. For example, prior to v7.6.0 firmware the minimum RAM recommendation for VMWare appliances was 4 GB. As of v7.6.0 the requirement was increased to 8GB.

DataPower firmware v7.5.0 requirements.
DataPower firmware v7.6.0 requirements.

P.S. The firmware upgrade procedure was updated accordingly.

Gosha

Friday, June 30, 2017

Transforming XG45 and XI52 Virtual Appliances to IDG

Are you stuck with XG45 / XI52? Want to benefit from new functionality available only with the new firmware versions (v7.6+), but unable to upgrade? Wish you had IDG instead? Well, the good news are that can have your old virtual appliances IDGed! Just follow the technote which outlines the process of transforming XG45 / XI52 to IDG.

Bibliography:
  1. IBM technote
  2. DeveloperWorks forum

Gosha

Tuesday, April 4, 2017

WebSphere DataPower Weekly Friday Webcast

Each week one of IBM experts provides an overview of a special topic on the IBM WebSphere DataPower platform.

Registration link - https://www-01.ibm.com/marketing/iwm/iwm/web/signup.do?source=swg-wdwfw

Wednesday, March 22, 2017

How to: connecting Mac to DataPower console using serial cable

Updates


Sep 30,2019

*** Not yet tested ***

After plugging in the serial-to-USB cable to the Mac, a new device should be added under /dev/
ls /dev/cu.usbserial-*

then you could connectusing:
screen /dev/device baud_rate

No additional software needed.

Feb 1,2019

It seems that the Serial app by decisivetactics provides same capabilities.

Original post

A few years ago I've moved to MacBook Pro as my working laptop. And from the very beginning I've faced the problem of connecting my Mac to DataPower serial port. And since I've been doing around 10-20 DataPower installation a year, you can imagine yourself that it was very annoying. It took me quite a while till I've found a solution (with a help from a friend that works at IBM). So here it is:
  1. Install PL203 driver for Mac from prolific site. That should be pretty straight forward.
  2. As a terminal (instead of PuTTY) I use SerialTools which can be downloaded from the App Store.
You might need to tweak the checkbox based configuration, however down the road it works and does a great job!

Gosha

Bibliography

  1. Connecting to the serial console of a DataPower Gateway Appliance
    https://render-prd-trops.events.ibm.com/support/pages/connecting-serial-console-datapower-gateway-appliance

Friday, November 20, 2015

How to download certificates from DataPower to your desktop

Following a conversation on a DataPower developerworks forum, I have decided to write a dedicated post in my blog.

DataPower is a highly secured device. Until recently I was under the impression, that once a digital certificate was uploaded to DataPower, it cannot be downloaded. Boy, I was wrong. The following section is a step by step guide to download certificates from a DataPower appliance:


  1. Log into WebGUI.
  2. Create a new Certificate object, name it "FOO_Cert" and point to a certificate file located in the cert or sharedcert folder.
  3. Go to "Crypto Tools" and switch to "Export Crypto Object" tab.
  4. In the "Object Type" combo make sure "Certificate" is selected.
  5. In the "Object Name" type "FOO_Cert", certificate object created in the 1st step.
  6. In the "Output File Name" type FOO_Cert.xml
  7. Click "Export Crypto Object" button.
  8. Go to File Management and download the FOO_Cert.xml file from temporary folder.
  9. Remove all XML tags, leave only the base64 part, and save the file.
  10. The certificate is in the file!

Tested on IDG Virtual Edition, XI52 physical appliance, and other appliances. Not tested on appliances with enabled Common Criteria.

It can also be achieved using SOMA API function, and in one of my projects we've created a GUI for downloading certificates from DataPower.

It is important to state that since there is nothing wrong having certificates and public keys freely accessible, this doesn't make DataPower vulnerable, especially hence one still needs credentials to log into the appliance.


Fell free to reach out with any question.

Gosha