Integration Patterns and Best Practices for Salesforce - Part 2

This is my second blog to explain Integration patterns and best practices.  In this I will cover remaining patterns.

In order to understand the patterns covered in previous blog, refer below link:
Integration Patterns and Best Practices for Salesforce - Part 1

If you are planning to appear for "SALESFORCE CERTIFIED INTEGRATION ARCHITECTURE DESIGNER" exam then below information will be helpful.

Note: I have summarized the information provided in Integration Patterns and Practices documentation provided by salesforce in this blog.

Batch Data Synchronization

Use this pattern if you want to import data into Salesforce and export data out of Salesforce, taking into consideration that these imports and exports can interfere with end-user operations during business hours, and involve large amounts of data.
Below are different scenarions which can utilize this pattern:

• Extract and transform accounts, contacts, and opportunities from the current CRM system and load the data into Salesforce (initial  data import).
• Extract, transform, and load customer billing data into Salesforce from a remote system on a weekly basis (ongoing).
• Extract customer activity information from Salesforce and import it into an on-premises data warehouse on a weekly basis (ongoing backup).      

Change data capture    

• If remote system is master

Leverage a third-party ETL tool that allows you to run change data capture against source data. The tool reacts to changes in the source data set, transforms the data, and then calls Salesforce Bulk API to issue DML statements. This can also be implemented using the Salesforce SOAP API.  

• If Salesforce system is master

If Salesforce is the data source then you can use time/status information on individual rows to query the data and  filter the target result set. This can be implemented by using SOQL together with SOAP API and the query() method, or by the using SOAP API and the getUpdated() method.

In case of using middleware, it is recommend that you create the control tables and associated data structures in an environment that the ETL tool has access. This provides adequate levels of resilience. Salesforce should be treated as a spoke in this process and the ETL infrastructure is the hub.
For an ETL tool to gain maximum benefit from data synchronization capabilities, consider the following:

• Chain and sequence the ETL jobs to provide a cohesive process.
• Use primary keys from both systems to match incoming data.
• Use specific API methods to extract only updated data.
• If importing child records in a master-detail or lookup relationship, group the imported data using its parent key at the source to avoid locking. For example, if you’re importing contact data, be sure to group the contact data by the parent account key so that maximum number of contacts for a single account can be loaded in one API call. Failure to group the imported data usually results in the first contact record being loaded and subsequent contact records for that account to fail in the context of the API call.
•  Any post-import processing, such as triggers, should only process data selectively.

Error Handling and Recovery

• Exporting data from SFDC

During read operation from salesforce, middleware should perform below operations

• Log the error
• Retry the read operation
• Terminate if unsuccessful
• Send a notification

• Importing data into SFDC

Handling—Errors that occur during a write operation via middleware can result from a combination of factors in the application (record locking errors). The API calls return a result set that consists of the information listed below. This information should be used to retry the write operation (if necessary).

• Record identifying information
• Success/failure notification
• collection of errors for each record

Security Considerations

• A Lightning Platform license is required to allow authenticated API access to the Salesforce API.
• It is recommended to use standard encryption to keep password access secure.
• Use the HTTPS protocol when making calls to the Salesforce APIs. You can also proxy traffic to the Salesforce APIs through an on-premises security solution, if necessary.

Timeliness

Timeline is not significant factor as these operations runs in background. Loading batches during business hours might result in some contention, resulting in either a user's update failing, or more significantly, a batch load (or partial batch load) failing.
For organizations that have global operations, it might not be feasible to run all batch processes at the same time because the system might continually be in use. Data segmentation techniques using record types and other filtering criteria can be used to avoid data contention in these cases.

Data Volumes

This pattern is mainly used for bulk data import and export.

Remote Call-In

This pattern is used when remote system wants to connect to salesforce and after authentication, want to update records in SFDC.
Below are different options available for this:

• SOAP API

Query, Create update or delete records and obtain metadata information from Salesforce

Salesforce provides two WSDLs for remote systems:

• Enterprise WSDL—Provides a strongly-typed WSDL that’s specific to a Salesforce organization.
• Partner WSDL—Contains a loosely-typed WSDL that’s not specific to a Salesforce organization. It deals with considering subject structure.

Security :- The client executing SOAP API must have a valid login and obtain a session to perform any API calls. The API respects object-level and field-level security configured in the application based on the logged in user’s profile.

Data Volume :- For bulk data operations (more than 500,000 records), use the REST-based Bulk API.

• REST API

Query, Create update or delete records and obtain metadata information from Salesforce

REST exposes resources (entities/objects) as URIs and uses HTTP verbs to define CRUD perations on these resources. Unlike SOAP, the REST API requires no predefined contract, utilizes XML and JSON for responses, and has loose typing. REST API is lightweight and provides a simple method for interacting with Salesforce. Its advantages include ease of integration and development, and it’s an excellent choice for use with mobile applications and Web 2.0 projects.

Security:-  We recommend that the remote system establish an OAuth trust for authorization.  It’s also possible to make REST calls with a valid session ID that might have been obtained by other means (for example, retrieved by calling SOAP API or provided via an outbound message).

We recommend that clients that call the REST API cache and reuse the session ID to maximize performance, rather than obtaining a new session ID for each call.

• Custom Webservices/Apex Rest classes

We can create custom webservices and provide WSDL to remote system so that they can consume it and call custom webservices methods. If we create Apex rest services, then remote system can directly call URI’s.

Custom webservices or Apex Rest services are usefull when you need to update multiple records related to different objects in single call as logic for complete transaction is controlled by developer.

• Bulk API

Bulk API is based on REST principles, and is optimized for loading or deleting large sets of data. It has the same accessibility and security behavior as REST API.

Bulk API allows the client application to query, insert, update, upsert, or delete a large number of records asynchronously by submitting a number of batches, which are processed in the background by Salesforce. In contrast, SOAP API is optimized for real-time client applications that update small numbers of records at a time.

Although SOAP API can also be used for processing large numbers of records, when the data sets contain hundreds of thousands to millions of records, it becomes less practical. This is due to its relatively high overhead and lower performance characteristics.

Error Handling and Recovery

Error handling needs to be implemented by remote system or middleware. Middleware or remote system should implement retry logic and also need to make sure that duplicate request is coming to salesforce. We can handle duplicate request in case of custom webservices or apex rest services but it is required for remote system to have some mechanism for this.

Timelines

SOAP and REST API’s are synchronous.

Data Volume

SOAP/REST API

• Login—The login request size is limited to 10 KB or less.
• Create, Update, Delete—The remote system can create, update, or delete up to 200 records at a time. Multiple calls can be made to process more than a total of 200 records, but each request is limited to 200 records in size.
• Query Results Size — By default, the number of rows returned in the query result object (batch size), returned in a query() or queryMore() call is set to 500. Where the number of rows to be returned exceeds the batch size, use the queryMore() API call to iterate through multiple batches. The maximum batch size is 2,000 records

BULK API
Bulk API is synchronous when submitting the batch request and associated data. The actual processing of the data occurs asynchronously in the background.

• Up to 2,000 batches can be submitted per rolling 24–hour period.
• A batch can contain a maximum of 10,000 records.

UI Update Based on Data Changes

When an event occurs in Salesforce like update to any record, user should be notified in the Salesforce user interface without having to refresh their screen and potentially losing work.
The recommended solution to this integration problem is to use the Salesforce Streaming API.
This solution is comprised of the following  components:

• A PushTopic with a query definition that allows you to:
  • Specify what events trigger an update
  • Select what data to include in the notification
• A JavaScript-based implementation of the Bayeux protocol (currently CometD) that can be used by the user interface
• A Visualforce page
• A JavaScript library included as a static resource

Benefit of Streaming API

• No need to write pooling mechanism to identify the records changes
• User does not have to refresh record or invoke any action to get latest updates

Limitations

• Delivery of notifications isn’t guaranteed.
• Order of notifications isn’t guaranteed.
• Notifications aren’t generated from record changes made by Bulk API.

Security Considerations

It respect Salesforce organization-level security.

Idempotent Design Considerations

• Remote Process Invocation—Request and Reply / Request and Forget

It’s important to ensure that the remote procedure being called is idempotent means it can identify if any repeated request is coming to avoid duplicates request processing. It’s almost impossible to guarantee that Salesforce only calls once, especially if the call is triggered from a user interface event. Even if Salesforce makes a single call, there’s no guarantee that other processes (for example, middleware) do the same.

The most typical method of building an idempotent receiver is for it to track duplicates based on unique message identifiers sent by the consumer. Apex web service or REST calls must be customized to send a unique message ID.

• Remote Call In

The remote system must manage multiple (duplicate) calls, in the case of errors or timeouts, to avoid duplicate inserts and redundant updates (especially if downstream triggers and workflow rules fire). While it’s possible to manage some of these situations within Salesforce (particularly in the case of custom SOAP and REST services), we recommend that the remote system (or middleware) manages error handling and idempotent design.

Hope this will help!!

Integration Patterns and Best Practices for Salesforce - Part 1

Whenever you have to integrate Salesforce with remote system, then you need to consider different integration scenarios and API’s availability. Integration patterns helps you identify the correct pattern or API to build robust framework for integration.

If you are planning to appear for "SALESFORCE CERTIFIED INTEGRATION ARCHITECTURE DESIGNER" exam then below information will be helpful.

In complex integration scenarios, mix of integration patterns is used by weighing pros and cons accordingly.

Today I am going to cover below patterns. For remaining patterns, refer below blog:
Integration Patterns and Best Practices for Salesforce - Part 2

Remote Process Invocation—Request and Reply

This pattern is used when you need to call remote system and wait for response. After getting response you want to update something in SFDC in synchronous manner. This pattern is used when you want to complete request and process the response in same transaction.

Different Scenarios:

• Onclick of button in UI :- In VF page, onclick of button you want to call remote system to get order information and display it to user on VF page without storing it in salesforce. You can implement this with Apex SOAP or REST callouts.

• Apex triggers :-You can use Apex triggers to perform callouts based on record data changes. All calls made from within the trigger context must execute asynchronously from the initiating event. Therefore, apex triggers isn’t recommended for this pattern.

• Batch Class :- You can make calls to a remote system from a batch job. This solution allows batch remote process execution and processing of the response from the remote system in Salesforce. However, a given batch has limits to the number of calls. So, batch class is also not recommended for this pattern.

Error Handling and Recovery

It’s important to include an error handling and recovery strategy as part of the overall solution.

• Error handling—When an error occurs (exceptions or error codes are returned to the caller), the caller manages error handling. For example, an error message displayed on the end-user’s page or logged to a table requiring further action.
• Recovery—Changes aren’t committed to Salesforce until the caller receives a successful response. For example, the order status isn’t updated in the database until a response that indicates success is received. If necessary, the caller can retry the operation.

Security Considerations

Any call to a remote system must maintain the confidentiality, integrity, and availability of the request. The following security considerations are specific to using Apex SOAP and HTTP calls in this pattern.

• One-way SSL is enabled by default (for apex callout and outbound messages), but two-way SSL is supported with both self-signed and CA-signed certificates to maintain authenticity of both the client and server.
• Salesforce does not currently support WS-Security.
• Where necessary, consider using one-way hashes or digital signatures using the Apex Crypto class methods to ensure request integrity.
• The remote system must be protected by implementing the appropriate firewall mechanisms.

Timeliness

Timeliness is of significant importance in this pattern. Usually:

• The request is typically invoked from the user interface, so the process must not keep the user waiting.
• Salesforce has a configurable timeout of up to 60 seconds for calls from Apex.
• Completion of the remote process is executed in a timely manner to conclude within the Salesforce timeout limit and within user expectations.

Data Volumes

This pattern is used primarily for small volume, real-time activities, due to the small timeout values and maximum size of the request or response for the Apex call solution. Do not use this pattern in batch processing activities in which the data payload is contained in the message.

Remote Process Invocation—Fire and Forget

This pattern is utilized when you want to send some information to remote system and don’t want wait for response. Consider the scenario in which order is created in salesforce and then order information is send to order fulfillment system (outside salesforce) which will process the order.
Best example for this pattern utilization is Outbound messages.

Different Scenarios:

• Outbound messages and call back

No customization is required in Salesforce to implement outbound messaging. The recommended solution for this type Workflow-driven outbound messaging Good of integration is when the remote process is invoked from an insert or update event. Salesforce provides a workflow-driven outbound messaging capability that allows sending SOAP messages to remote systems triggered by an insert or update operation in Salesforce. These messages are sent asynchronously and are independent of the Salesforce user interface. The outbound message is sent to a specific remote endpoint. The remote service must be able to participate in a contract-first integration where Salesforce provides the contract. On receipt of the message, if the remote service doesn’t respond with a positive acknowledgment, Salesforce retries sending the message, providing a form of guaranteed delivery.

A single outbound message can send data only for a single object. A callback can be used to retrieve data from other related records, such as related lists associated with the parent object. The outbound message provides a unique SessionId that you can use as an authentication token to authenticate and authorize a callback with either the SOAP API or the REST API. The system performing the callback isn’t required to separately authenticate to Salesforce. The standard methods of either API can then be used to perform the desired business functions. A typical use of this variant is the scenario in which Salesforce sends an outbound message to a remote system to create a record. The callback updates the original Salesforce record with the unique key of the record created in the remote system.

This is best use case for this pattern as information will be send from salesforce using outbound message without any customization. If other system is unavailable then, outbound has in-built retry mechanism for 24 hours.

Also we can pass session id of user so that remote system can perform callback to SFDC to update records providing confirmation that request have been successfully received by remote system.

Given the static, declarative nature of the outbound message, no complex integration such as aggregation, orchestration, or transformation, can be performed in Salesforce. The remotesystem or middleware must handle these types of operations .

• VF and custom controllers

You can utilize VF page and controllers to send information to remote system but it involves customization to grantee delivery of messages as remote system can be have downtime when request is send.

• Apex triggers/Batch Class

You can use Apex triggers to perform callouts based on record data changes. All calls made from within the trigger context must execute asynchronously from the initiating event. Batch and triggers also have API call limits

Error Handling and Recovery

An error handling and recovery strategy must be considered as part of the overall solution. The best method depends on the solution you choose.

• Apex Callout – We need to maintain logic for retry and error handling if we didn’t get acknowledgement from remote or gets timeout error. Recovery mechanism is complex in this scenario
• Outbound messages – Outbound has inbuilt retry mechanism.

Security Considerations

A call to a remote system must maintain the confidentiality, integrity, and availability of the request.For Outbound message, Whitelist Salesforce server IP ranges for remote integration servers as all Outbound messages are delivered from Salesforce IP addresses.

Timeliness

Timeliness is less of a factor with the fire-and-forget pattern. Control is handed back to the client either immediately or after positive acknowledgment of a successful hand-off in the remote system. With Salesforce outbound messaging, the acknowledgment must occur within 24 hours; otherwise, the message expires.

Data Volumes

Data volume considerations depend on which solution you choose. There is no limit on number of outbound messages sent but limits are there for apex callouts.

Note:- I have summarized the information provided in Integration Patterns and Practices documentation provided by salesforce in this blog.

Hope this will help!!!

Lightning Treegrid : Generic Component to display hierarchy in tabular format for any sobject

In this blog, I am going to share a reusable component which can be used to display hierarchy of records along with other fields information in treegrid format.

Previously we have to use jquery to implement this but now "Lightning:treegrid" tag have been introduced which can be used to display treegrid in lightning framework.

In order to use this component, you have to pass below parameters to this components:

• ltngcurrentRecId =  RecordId (15 or 18 digit)
• ltngSobjectName = Object API Name
• ltngParentFieldAPIName = Parent Field API name which create self relationship
• ltngColumnLabelList = Specify the column label
• ltngColumnAPINameList = Specify the API field names in same order as that of column
• ltngHyperlinkColumn = Field API Name which will work as hyperlink for record detail page

Suppose you have display treegrid for Account hierarchy by using below syntax:

<c:SK_GenericTreeGridCmp ltngcurrentRecId="0019000000ld4kS"
ltngSobjectName="Account"
ltngParentFieldAPIName="ParentId"
ltngColumnLabelList="['Name','Type','Industry','Account Owner']"
ltngColumnAPINameList="['Name','Type','Industry','Owner.Name']"
ltngHyperlinkColumn="Name"/>

Below is output:

Suppose you have to display case hierarchy, then use below code snippet:

<c:SK_GenericTreeGridCmp ltngcurrentRecId="5009000000GJkJE"
ltngSobjectName="Case"
ltngParentFieldAPIName="ParentId"
ltngColumnLabelList="['CaseNumber','Subject','Status','Case Owner']"
ltngColumnAPINameList="['CaseNumber','Subject','Status','Owner.Name']"
ltngHyperlinkColumn="CaseNumber"
ltngHeaderValue="Case Hierarchy"/>

Below is output snapshot

Below is complete code snippet for your reference:

You can download the code from GitHub by using below URL:

Lightning-Treegrid-Generic-Component-to-display-hierarchy-in-tabular-format-for-any-sobject

Note:

• Specify Field API properly as javascript is case sensitive. For example, specify "Name" instead of "name"
• For adding parent field API names, provide API names properly. For example for Account owner use, "Owner.Name" instead of "owner.name".

Hope this will help!!!

Lightning Components Tutorial : Easy way to become Lightning developer

In this blog, I am going to share all lightning components concept which is required for Lightning developer starting from basics to advance topics.

You can learn lightning by following below blogs related to lightning.

1. Lightning Component Basics: Lightning Component Framework
2. Lightning Components Basics : Attributes, Expressions and Value Providers
3. Lightning Components Basics : Handling Actions in Controller.js
4. How to set or get picklist (drop down) values in Lightning Components
5. Lightning Components Basics : Considerations while calling server-side controller actions
6. Application Events : Way to Communicate Between Different Lightning Components
7. Component Events: Way to Communicate Between Components in Containment Hierarchy
8. Raising and Handling Custom Events in Salesforce Lightning
9. Salesforce Lightning Events : Hands on Training Guide
10. Salesforce 1 Events in Lightning Experience and Salesforce 1 app
11. Lightning Data Services : Way to perform operation on records without using server-side Apex class
12. Lightning Data Services : Hands on Training Guide
13. Creating Lightning Components Dynamically
14. Dynamically Creating and Destroying Lightning Button(ui:button)
15. Adding Lightning Components in VF Page and Passing Values from URL
16. Inheritance in Lightning Components
17. Firing Event from Lightning Component and Passing Parameter to VF Page
18. How to Fire Lightning Events from VF Page in Lightning
19. Custom Component to show Loading Spinner in Lightning during Server or Client side Operations
20. Why to use design resource in Lightning bundle and how to add dynamic Options in datasource for Design Component
21. Formatting and Localizing dates in javascript in Lightning
22. Using Chartjs in Lightning Components
23. Lightning Tree : Generic Lightning Component to Display Hierarchy for any sObject
24. Lightning:treegrid - Displaying Account hierarchy using treegrid
25. Aura.Action : Lightning Framework Specific Attributes Type
26. Aura.Component[] : Lightning Framework Specific Attributes Type

Hope this will help!!

Aura.Action : Lightning Framework Specific Attributes Type

In this blog, I am going to explain "Aura.Action" attribute type and its usage.

There is another Lightning Framework specific attribute "Aura.Component[]". If you want to learn more about this then refer below blog:

Aura.Component[] : Lightning Framework Specific Attributes Type


Aura.Action

Use this type of attribute if you want to pass the reference of parent lightning component controller.js function to nested/child lightning component. This will help to invoke parent component JS function from child component markup.

Below is syntax to define this kind of attribute:

<aura:attribute name="parentJSFunction" type="Aura.Action"/>

This is useful if you want to make reusable component to appear based on values passed from parent and close/hide the component by invoking parent lightning component JS function.

I have created a sample component to explain how to invoke parent component function and child component function by passing "Aura.Action" attribute to child component.

Below is snapshot of Lightning component ("SK_AuraActionDemoCmp")  UI which contains child component ("SK_AuraActionChildCmp")  inside it.

Child component contains 2 buttons which will invoke JS function of parent and same component. Child component contains an attribute of type "Aura.Action" through which we will parent parent component JS function reference.

<aura:attribute name="closeChildDivFunction" type="Aura.Action" default="{!c.defaultCloseAction}"/>

If you don't pass JS function from parent component then child component will invoke its own JS function "defaultCloseAction".

If user click on these 2 buttons, then below things will happen:

• If user clicks on "Close via Parent JS function" button,then complete content will also get removed and ltnguserActionMessage will display message saying "Close via Parent JS function button clicked. Please refresh your browser if you again want to load this demo component"

• If user clicks on "Close via ChildCmp JS function" button,then only child component body will be removed and ltnguserActionMessage  will display message saying "Close via ChildCmp JS function button clicked.Please refresh your browser if you again want to load this demo component"

Please find below complete code reference:

Hope this will help!!

Lightning:treegrid - Displaying Account hierarchy using treegrid

Salesforce has introduced new tag called lightning:treegrid which can be used to display tree structure in tabular form. Lightning:tree can be used to display the hierarchy but if you want to display additional information then lightning:treegrid becomes useful.

If you want to lean how to use lightning:tree tag to display hierarchy for any record then refer below URL:

Lightning Tree : Generic Lightning Component to Display Hierarchy for any sObject

If you have to display account hierarchy along with another fields information as displayed in below image, then use treegrid.

Below is complete code to display account hierarchy. Just pass account record Id and lightning component will display complete hierarchy.


Hope this help!!!