Wednesday, January 3, 2018

XSLT for XML to JSON transformation in WSO2 ESB

Introduction

You might be running in problems when ever you want to transform a xml payload to a respective json payload by having the freedom to manipulate the json element types. ex: you might want to keep a json element as a json array even it's only consists with a single json object. Another instance would be when you want to have the type as a String for an ID which contains only numbers, then what ESB would do is convert it right way to a json number. 

In those similar instances you might want to do some customization to have it in your way. one method is to develop a XSLT to do the transformation. 

Please see the following XSLT which I have been using for the requirement and I will explain what each part of the transformation does.

<xsl:stylesheet
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:json="http://json.org"
 version="1.0">

 <xsl:output method="text" indent="yes" encoding="UTF-8"/>
 <xsl:strip-space elements="*"/>

 <xsl:param name="skip-root" select="false()"/>

 <xsl:template match="/">
  <xsl:text>{</xsl:text>
  <xsl:choose>
   <xsl:when test="$skip-root = 'true'">
    <xsl:apply-templates select="child::node()/child::node()"/>
   </xsl:when>
   <xsl:otherwise>
    <xsl:apply-templates select="child::node()"/>
   </xsl:otherwise>
  </xsl:choose>
  <xsl:text>}</xsl:text>
 </xsl:template>

 <xsl:template match="@* | *">
  <xsl:if test="preceding-sibling::*">
   <xsl:text>,</xsl:text>
  </xsl:if>
  <xsl:choose>
   <xsl:when test="parent::*/@json:type = 'array'"/>
   <xsl:otherwise>
    <xsl:text>"</xsl:text>
    <xsl:value-of select="local-name()"/>
    <xsl:text>"</xsl:text>
    <xsl:text>:</xsl:text>
   </xsl:otherwise>
  </xsl:choose>
  <xsl:choose>
   <xsl:when test="@json:type = 'array'">
    <xsl:text>[</xsl:text>
    <xsl:apply-templates select="child::*"/>
    <xsl:text>]</xsl:text>
   </xsl:when>
   <xsl:when test="@json:type = 'object' or count(attribute::*[not(namespace-uri() = 'http://json.org') and not(namespace-uri() = 'http://www.w3.org/2001/XMLSchema-instance')] | child::*) > 0">
    <xsl:text>{</xsl:text>
    <xsl:for-each select="attribute::*[not(namespace-uri() = 'http://json.org') and not(namespace-uri() = 'http://www.w3.org/2001/XMLSchema-instance')] | child::*">
     <xsl:if test="not(preceding-sibling::*) and position() != 1">
      <xsl:text>,</xsl:text>
     </xsl:if>
     <xsl:apply-templates select="current()"/>
    </xsl:for-each>
    <xsl:text>}</xsl:text>
   </xsl:when>
   <xsl:otherwise>
    <xsl:call-template name="value"/>
   </xsl:otherwise>
  </xsl:choose>
 </xsl:template>

    <xsl:template name="value">
     <xsl:variable name="value">
  <xsl:value-of select="text() | ."/>
 </xsl:variable>
     <xsl:choose>
      <xsl:when test="not(normalize-space($value))">
   <xsl:when test="@xsi:nil = 'true'">
    <xsl:text>null</xsl:text>
   </xsl:when>
   <xsl:otherwise>
    <xsl:text>""</xsl:text>
   </xsl:otherwise>
      </xsl:when>
      <xsl:when test="@json:type = 'string'">
       <xsl:text>"</xsl:text>
   <xsl:value-of select="$value"/>
       <xsl:text>"</xsl:text>
      </xsl:when>
      <xsl:when test="string(number($value)) != 'NaN' ">
       <xsl:value-of select="$value"/>
      </xsl:when>
  <xsl:when test="translate($value, 'TRUE', 'true') = 'true'">
   <xsl:text>true</xsl:text>
  </xsl:when>
  <xsl:when test="translate($value, 'FALSE', 'false') = 'false'">
   <xsl:text>false</xsl:text>
  </xsl:when>
  <xsl:otherwise>
       <xsl:text>"</xsl:text>
   <xsl:value-of select="$value"/>
       <xsl:text>"</xsl:text>
  </xsl:otherwise>
     </xsl:choose>
    </xsl:template>
</xsl:stylesheet>


You can specify with the param "skip-root" whether you want to skip that the wrapping element. This is basically needed when you are given a payload wrapped with a <jsonObject>.

Here the namespace json has been used to identify json data types for each attribute should be represented. and the xsi namespace has been used to force to have null values when the "nil=true" value is there as an attribute. Otherwise the default value is for the empty element would be an "" (empty) string.

Rest was designed to automatically handle the attributes and child elements the same and populate the json payload as necessary.

To use this xslt you might want to use the following with your sequence after the transformation.

<payloadFactory media-type="json">
<format>$1</format>
<args><arg evaluator="xml" expression="$body/*[local-name()='text']"/></args>
</payloadFactory>

Please remember to set the messageType property to "application/json" too.

Saturday, August 12, 2017

OSGi Service Trackers

Requirement is to have a dynamic mapping with an interface implementation provided by the main OSGi bundle. At a given time there could only be a default and a single custom implementation. For this purpose I'm using OSGi ServiceTracker to dynamically assign the implementation.


Use case:

This interface "org.siriwardana.sample.core.MessageHandlerFactory" will be exported by the core bundle and will be implemented by the default bundle and a custom implementation.

/**
 *
 * Interface for message handler factory. Custom deployments should implement this interface
 */
public interface MessageHandlerFactory {

    MessageHandler getHandler(String messageType);
}

Following is the org.siriwardana.sample.core.MessageHandler interface which is also exported by the core bundle.


/**
 *
 * Custom message handler which should be implemented by the custom deployment bundle to handle
 */
public interface MessageHandler {

    /**
     *Create the message with a custom implementation.
     * @return String
     */
    String createReqMsg();

    /**
     *
     * Handle response of the request as per the custom implementation.
     */
    void handleResponse (String response);

    /**
     *
     * Handle error as per the custom implementation
     */
    void onError(Exception e);

}

Default service bundle and a custom implementation service bundle will be available at runtime and the custom implementation will get the priority by the consumer bundle.

Solution:
ServiceTracker (org.osgi.util.tracker.ServiceTracker) and ServiceTrackerCustomizer (org.osgi.util.tracker.ServiceTrackerCustomizer) will be used to dynamically used by the consumer bundle.

Following bundle activator implementation will demonstrates the solution. 

/**
 * @scr.component name="org.siriwardana.sample.consumer" immediate="true"
 */
public class ServiceComponent {

    private static Log LOGGER = LogFactory.getLog(ServiceComponent.class);
    private static final String MESSAGE_HANDLER_DEFAULT = "default";

    private ServiceTracker serviceTracker;
    private BundleContext bundleContext;
    private ServiceRegistration defaultHandlerRef;

    @SuppressWarnings("unchecked")
    protected void activate(ComponentContext context) {
        bundleContext = context.getBundleContext();

        Dictionary<String, String> props = new Hashtable<>();
        props.put(Constants.MESSAGE_HANDLER_KEY, MESSAGE_HANDLER_DEFAULT);

        if (bundleContext != null) {
           
            ServiceTrackerCustomizer trackerCustomizer = new Customizer();
            serviceTracker = new ServiceTracker(bundleContext, MessageHandlerFactory.class.getName(), trackerCustomizer);
            serviceTracker.open();
            LOGGER.debug("ServiceTracker initialized");
        } else {
            LOGGER.error("BundleContext cannot be null");
        }
    }

    protected void deactivate(ComponentContext context) {

        defaultHandlerRef.unregister();
        serviceTracker.close();
        serviceTracker = null;
        LOGGER.debug("ServiceTracker stopped. Cloud Default handler bundle deactivated.");
    }

    private void setMessageHandlerFactory(ServiceReference<?> reference) {

        MessageHandlerFactory handlerFactory = (MessageHandlerFactory) bundleContext.getService(reference);
        LOGGER.debug("MessageHandlerFactory  is acquired");
        ServiceDataHolder.getInstance().setHandlerFactory(handlerFactory);
    }

    private void unsetMessageHandlerFactory(MessageHandlerFactory handlerFactory) {

        LOGGER.debug("MessageHandlerFactory  is released");
        ServiceDataHolder.getInstance().setHandlerFactory(null);
    }

    /**
     *
     * Service tracker for Message handler factory implementation
     */
    private class Customizer implements ServiceTrackerCustomizer {

        @SuppressWarnings("unchecked")
        public Object addingService(ServiceReference serviceReference) {

            LOGGER.debug("ServiceTracker: service added event invoked");
            ServiceReference serviceRef = updateMessageHandlerService();
            return bundleContext.getService(serviceRef);
        }

        public void modifiedService(ServiceReference reference, Object service) {
            LOGGER.debug("ServiceTracker: modified service event invoked");
            updateMessageHandlerService();
        }

        @SuppressWarnings("unchecked")
        public void removedService(ServiceReference reference, Object service) {

            if (reference != null) {
                MessageHandlerFactory handlerFactory = (MessageHandlerFactory) bundleContext.getService(reference);
                unsetMessageHandlerFactory(handlerFactory);
                LOGGER.debug("ServiceTracker: removed service event invoked");
                updateMessageHandlerService();
            }
        }

        private ServiceReference updateMessageHandlerService() {

            ServiceReference serviceRef = null;
            try {
                ServiceReference<?>[] references = bundleContext
                        .getAllServiceReferences(MessageHandlerFactory.class.getName(), null);
                for(ServiceReference<?> reference : references) {
                    serviceRef = reference;
                    if (!MESSAGE_HANDLER_DEFAULT
                            .equalsIgnoreCase((String) reference.getProperty(Constants.MESSAGE_HANDLER_KEY))) {
                        break;
                    }
                }
                if (serviceRef != null) {
                    LOGGER.debug("ServiceTracker: HandlerFactory updated. Service reference: " + serviceRef);
                    setMessageHandlerFactory(serviceRef);
                } else {
                    LOGGER.debug("ServiceTracker: HandlerFactory not updated: Service reference is null");
                }
            } catch (InvalidSyntaxException e) {
                LOGGER.error("ServiceTracker: Error while updating the MessageHandlers. ", e);
            }
            return serviceRef;
        }
    }
}

Following is the custom implementation bundle which will register it's service for the MessageHandlerFactory interface.


/**
 * @scr.component name="org.siriwardana.custom" immediate="true"
 */
public class CustomMessageHandlerFactoryComponent {

    private static final String MESSAGE_HANDLER = "CUSTOM";
    private static Log LOGGER = LogFactory.getLog(CustomMessageHandlerFactoryComponent.class);

    private ServiceRegistration serviceRef;

    protected void activate(ComponentContext context) {

        BundleContext bundleContext = context.getBundleContext();
        Dictionary<String, String> props = new Hashtable<>();
        props.put(Constants.MESSAGE_HANDLER_KEY, MESSAGE_HANDLER);

        serviceRef = bundleContext.registerService(MessageHandlerFactory.class, new CustomMessageHandlerFactory(), props);
        LOGGER.debug("Custom Message handler impl bundle activated ");
    }

    protected void deactivate(ComponentContext context) {
        serviceRef.unregister();
        LOGGER.debug("Custom Message handler impl bundle deactivated ");
    }
}

When every an Interface implementation is available, ServiceTracker will update the consumer bundle.

Sunday, July 31, 2016

WSO2 DSS calling Stored Procedure with a UDT


In a case where you have to query a Oracle stored procedure (SP) from WSO2 Data services which contains a  User Defined Types (UDT); you'll have to define the data services definition as follows.

As an example please see the following.

SP (details) with a UDT (m_numbers) defined globally. under database name "clas"

create or replace TYPE  m_numbers AS VARRAY(20) OF VARCHAR2(10);

create or replace PACKAGE  details AS

      PROCEDURE user
      (
            in_name         IN   VARCHAR2,
            in_count        IN   NUMBER,
            out_numbers    OUT  m_numbers,
            out_code       OUT  VARCHAR2,
            out_message    OUT  VARCHAR2
      );
    
    END details;


Then your WSO2 dataservice should be defined as follows.

<data name="testService" transports="http https local">
   <config id="default">
      <property name="driverClassName">oracle.jdbc.driver.OracleDriver</property>
      <property name="url">jdbc:oracle:thin:@//localhost:1521/XE</property>
      <property name="username">xxxx</property>
      <property name="password">xxxx</property>
   </config>
   <query id="getMobileNumberQuery" useConfig="default">
      <sql>call CLAS.DETAILS.USER(:in_name, :in_count, :out_numbers, :out_code, :out_message)</sql>
      <result element="Numbers" rowName="Number">
         <element column="out_code" name="out_code" xsdType="string"/>
         <element column="out_message" name="out_message" xsdType="string"/>
         <element arrayName="m_numbers" column="out_numbers" name="out_numbers" xsdType="string"/>
      </result>
      <param name="in_name" sqlType="STRING"/>
      <param name="in_count" sqlType="INTEGER"/>
      <param name="out_code" sqlType="STRING" type="OUT"/>
      <param name="out_message" sqlType="STRING" type="OUT"/>
      <param name="out_numbers" paramType="ARRAY" sqlType="ARRAY" structType="M_NUMBERS" type="OUT"/>
   </query>
   <operation name="getMobileNumber">
      <call-query href="getMobileNumberQuery">
         <with-param name="in_name" query-param="in_name"/>
         <with-param name="in_count" query-param="in_count"/>
      </call-query>
   </operation>
</data>

If your element definition is wrong you'll be probably end up with the following exception.


[2016-07-24 17:46:03,446] ERROR - SQLQuery DS Fault Message: Error in 'createProcessedPreparedStatement'
DS Code: UNKNOWN_ERROR
Nested Exception:-
java.sql.SQLException: invalid name pattern: clas.details.user

DS Fault Message: Error in 'createProcessedPreparedStatement'
DS Code: UNKNOWN_ERROR
Nested Exception:-
java.sql.SQLException: invalid name pattern: clas.details.user

        at org.wso2.carbon.dataservices.core.description.query.SQLQuery.createProcessedPreparedStatement(SQLQuery.java:1602)
        at org.wso2.carbon.dataservices.core.description.query.SQLQuery.processPreStoredProcQuery(SQLQuery.java:933)
        at org.wso2.carbon.dataservices.core.description.query.SQLQuery.runPreQuery(SQLQuery.java:2303)
        at org.wso2.carbon.dataservices.core.description.query.Query.execute(Query.java:275)
        at org.wso2.carbon.dataservices.core.engine.CallQuery.executeElement(CallQuery.java:188)


[1] http://prabathabey.blogspot.com.au/2012/05/query-udtsuser-defined-types-with-wso2.html



Tuesday, February 9, 2016

Configure SSL Protocol versions in Carbon servers


Both the server and client can support multiple TLS versions. During the negotiation phase they come to a mutual agreement on the highest protocol version each supports and communicate over that. In your deployment you might want to configure the SSL/TLS protocol versions that the carbon servers support. You can do that as follows.

Lets describe this as in [1] which is written by Prabath Siriwardena when disabling SSLv3.

Open [product_home]/repository/conf/tomcat/catalina-server.xml
  1. Find the Connector configuration corresponding to TLS - usually this is having the port as 9443 and "sslProtocol" as TLS.
  2. If you are using JDK 1.6 then remove the attribute sslProtocol="TLS" from the above configuration and replace it with: sslEnabledProtocols="TLSv1"
  3. If you are using JDK 1.7 then remove the attribute sslProtocol="TLS" from the above configuration and replace it with: sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2"
If you have enabled pass-thru transport in any WSO2 product (ESB, API Manager) - you also need to do the following configuration change.

Open [product_home]/repository/conf/axis2/axis2.xml
Find the transportReceiver configuration element for org.apache.synapse.transport.passthru.PassThroughHttpSSLListener
  1. If you are using JDK 1.6 - add the following parameter under transportReceiver. <parameter name="HttpsProtocols">TLSv1</parameter> 
  2. If you are using JDK 1.7 - add the following parameter under transportReceiver. <parameter name="HttpsProtocols">TLSv1,TLSv1.1,TLSv1.2</parameter> 
Following explains how to validate the fix. You can download TestSSLServer.jar from here.

$ java -jar TestSSLServer.jar localhost 9443


To test the pass-thru transport use the following command with the corresponding port.


$ java -jar TestSSLServer.jar localhost 8243


Output before the fix

Supported versions: SSLv3 TLSv1.0
Deflate compression: no
Supported cipher suites (ORDER IS NOT SIGNIFICANT):
SSLv3
RSA_EXPORT_WITH_RC4_40_MD5
RSA_WITH_RC4_128_MD5
RSA_WITH_RC4_128_SHA
RSA_EXPORT_WITH_DES40_CBC_SHA
RSA_WITH_DES_CBC_SHA
RSA_WITH_3DES_EDE_CBC_SHA
DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
DHE_RSA_WITH_DES_CBC_SHA
DHE_RSA_WITH_3DES_EDE_CBC_SHA
RSA_WITH_AES_128_CBC_SHA
DHE_RSA_WITH_AES_128_CBC_SHA
RSA_WITH_AES_256_CBC_SHA
DHE_RSA_WITH_AES_256_CBC_SHA
(TLSv1.0: idem)

Output after the fix

Supported versions: TLSv1.0
Deflate compression: no
Supported cipher suites (ORDER IS NOT SIGNIFICANT):
TLSv1.0
RSA_EXPORT_WITH_RC4_40_MD5
RSA_WITH_RC4_128_MD5
RSA_WITH_RC4_128_SHA
RSA_EXPORT_WITH_DES40_CBC_SHA
RSA_WITH_DES_CBC_SHA
RSA_WITH_3DES_EDE_CBC_SHA
DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
DHE_RSA_WITH_DES_CBC_SHA
DHE_RSA_WITH_3DES_EDE_CBC_SHA
RSA_WITH_AES_128_CBC_SHA
DHE_RSA_WITH_AES_128_CBC_SHA
RSA_WITH_AES_256_CBC_SHA
DHE_RSA_WITH_AES_256_CBC_SHA

This all works fine unless you wants to disable TLSv1 for Carbon servers in JDK 6 or in JDK 7. For JDK 6 this can be ignored since mostly this will not be used spawn carbon instances now.
For JDK 7 this will fail even for login since it uses httpclient for stubs and when it comes as a client Carbon servers support all TLS versions that JVM offer and there is no configuration to disable client supported TLS versions. But why it's failing for JDK 7 is by default, it only enables TLSv1 for client connections [2][3]. And we cannot disable "TLSv1" (using -Dhttps.protocols=TLSv1.1,TLSv1.2 which is the only option) for client connections at JVM level for jre 7. New option (-Djdk.tls.client.protocols=TLSv1.1,TLSv1.2) is available from jre 8 [3] and also TLSv1.2 is the default enable version for jre 8, So that won't be a problem if we are using jre 8.


[1] http://wso2.com/library/blog-post/2014/10/blog-post-poodle-attack-and-disabling-ssl-v3-in-wso2-carbon-4.2.0-based-products/
[2] https://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#SunJSSEProvider
[3] https://blogs.oracle.com/java-platform-group/entry/diagnosing_tls_ssl_and_https

Sunday, January 31, 2016

Monetization solution for an API Management system

Introduction


In brief API Monetization with WSO2 API Manager is billing subscribers of a tenant. But WSO2 API Manager doesn't support out of the box for this. Reason is there would be tons of parameters to tackle if the product is supported out of the box for this. ex: billing rules, payment gateways and etc. 
So API Manager provides extension points to develop your own monetization solution. In this article I'll provide some brief description on those extension points and the available feature options you need to know

Monetization feature


This monetization feature is defined on API throttling tiers. Basically Subscribers will get billed according to the throttling tier they have subscribed to a particular API. So if your tenant admin wants to enable monetization for his/her tenant followings are options he should keep in mind to configure.

Defining Throttiling Tiers

API Manager previous versions, tenant had to define the tiers by editing tiers.xml in the registry. From API Manager 1.10 release this can be done from the admin dashboard [1]. The important thing I want to highlight here is following fields.

Stop On Quota Reach
This indicates the action to be taken when a user goes beyond the allocated quota. If checked, the user's requests will be dropped and an error response (HTTP Status code 429) will be given. If unchecked, the requests will be allowed to pass through.

Billing Plan
This is where you define whether this is Paid or a Free plan. In the development perspective of the underlying billing engine developer will use this to differentiate the workflow for the API subscription.

Description
Tenant admin should keep in mind to add a comprehensive description about the tier, because this is the only information will be shown at the API subscription. Otherwise subscriber might not have enough information to select an appropriate plan.          


Enable Monetization

First you have to enable monetization in API Manager. By enabling tenant's APIs in the store would get labeled as Paid, Free and Freemium. These labels are applied to those APIs depending on the tiers they are published on.  API throttling tier billing plans are either Paid or Free. If and API is published in tiers with a composition of those billing plans that API would be labeled as a Freemium, otherwise it would be labeled as either Paid or Free. 


Tenant admin can enable monetization as in the document [2]

Now you are good to go from setting up the flags for a tenant to use monetization. Next you want to engage your billing engine to API Manager to do metering and billing the subscribers of a tenant. 

To do this your system should have additional UIs/ functionality to enable monetization these should be developed externally. When a tenant enables monetization from your UIs, that particular tenant's default workflows should get replaced with your own Workflow executors. Process of developing your own workflows will be described later in this article. 
Replacement happen by calling the registry admin service (org.wso2.carbon.registry.core.service.RegistryService) and tenant registry loader admin service (org.wso2.carbon.registry.core.service.TenantRegistryLoader). You just have to replace it with your custom workflow executor configurations. [3][4] describes workflow extensions you can create and what are configurations you should add.
For an example say you have developed a custom API subscription workflow then you'll have to replace the following configuration


<SubscriptionCreation executor="org.wso2.carbon.apimgt.impl.workflow.SubscriptionCreationWSWorkflowExecutor">
    <Property name="serviceEndpoint">http://localhost:9765/services/SubscriptionApprovalWorkFlowProcess/</Property>
    <Property name="username">admin</Property>
    <Property name="password">admin</Property>
    <Property name="callbackURL">https://localhost:8243/services/WorkflowCallbackService</Property>
</SubscriptionCreation>

with your own API subscription workflow executor.

<SubscriptionCreation executor="org.wso2.carbon.cloud.monetization.apimgt.workflows.SubscriptionCreationWorkflowExecutor">
    <Property name="serviceEndpoint">https://milestones.appfactory.wso2.com:9443/services/APICloudMonetizationService/</Property>
    <Property name="username">rajith.siriw.ardana.gmail.com@mustanggt350</Property>
    <Property name="password">Admin</Property>
</SubscriptionCreation>

in registry location /_system/governance/apimgt/applicationdata/workflow-extensions.xml

Workflow executors 


When you are developing an API Management solution you might want to customize the API subscription flow, API deletion flow and the Application deletion flow to fit your requirements.
For an example say if you want to bill your subscribers and you want those information collected when they are subscribing to an API and etc. Best way to tackle those scenarios would be to use the extension points provided by API Manager which are called workflows. You just have to write your own workflows to replace the default subscription, subscription deletion, application deletion and etc workflows. You can find how to write workflow from the document [4]

Additionally for the document [4], when you are developing your own workflow executors, HTTP redirect to your pages residing in API Manager store and there is no domain change

  • You'll have to set the redirection confirmation message to null. Otherwise null message will pop up. this is a bug in API Manager 1.10.
  • You might not want to resume the workflow using the REST/SOAP endpoints described in [5] after your flow is executed. In that case you can use jaggery workflow modules resumeworkflow method.

References


[1] https://docs.wso2.com/display/AM1100/Defining+Throttling+Tiers
[2] https://docs.wso2.com/display/AM1100/Configuring+API+Monetization+Category+Labels
[3] https://docs.wso2.com/display/AM1100/Managing+Workflow+Extensions
[4] https://docs.wso2.com/display/AM1100/Configuring+HTTP+Redirection+for+Workflows
[5] https://docs.wso2.com/display/AM1100/Invoking+the+API+Manager+from+the+BPEL+Engine