Thursday, July 5, 2018

D365 Application Insights – Plug-ins & Custom Workflows

Not too many people were probably using Application Insights directly in plug-ins and custom workflow steps since it was introduced because every example I ever saw was writing using set a set of assemblies to making interfacing with the service easy (much like we do with D365). Of course awhile back I tried to ILMerge those assemblies into a plug-in but needless to say that didn’t work. I know a few instances where individuals had the logging proxied through something like an Azure Function which I’m sure worked perfectly well but then also took on the extra dependency (and cost) of whatever was doing the relaying. Luckily now we can avoid doing all that because we’ve now established there is a REST API and we can call it from just about anywhere.

I put a wrapper around it using C# so that you can easily include it plug-ins and workflow code. It’s ready to use right away but of course it can be modified as you see fit.
I’ve built in these telemetry types:
  • Trace
  • Event
  • Metric
  • Dependency
  • Exception
For examples of setting up and using, see the GitHub site’s wiki page.
By default with each request it’s going to log the following D365/CRM specific data as custom dimensions:
  • ExecutionContext
    • InitiatingUserId
    • ImpersonatingUserId
    • CorrelationId
    • Message
    • Mode
    • Depth
    • InputParameters
    • OutputParameters
    • Stage (plug-in)
    • WorkflowCategory (workflow)
    • PrimaryEntityName
    • PrimaryEntityId
    • OrganizationName
  • Organization version
  • Source (Plug-in or Workflow)
This is in addition to the normal items that are logged like the date/time of the event, originating location (datacenter city & state), etc.. Again it’s using the ids rather than friendly names to reduce overhead a bit. This could easily be updated to pull the friendly name or else you can cross reference it back to D365/CRM later.

Logging traces, metrics, events, caught exceptions, and dependencies are all pretty straight forward. Examples for everything is on the wiki page.

The constructor of the logger will need a configuration passed to it in terms of an object or a JSON string so you’ll need to have a way to get that information into your code by using the plug-ins configurations, workflow inputs, etc. You can control the types of telemetry logged, percentages of the events to log, or to enable and disable context tracking.

Besides context values if you so choose, nothing logged will be automatic. You’ll need to incorporate the Application Insights logger into your own exception handling and performance tracking.

The other important point to bring up is the actual duration it takes to log something. On average from D365 Online it was taking ~700ms to get a response back that the item was accepted. Doesn’t sound like much but if your plug-in is already busy, this isn’t going to help. The first way to mitigate that is to only log a percentage of “normal”, non-exception events. This was one of the driving factors to build in the functionality right away. That operates at a broad level across the logging instance and telemetry type. You can always add it your own filters to reduce the number of normal results. For example if you are logging the duration it takes to call an external web service and it normally take 50ms to complete, wrap the logging statement in something that checks to see if it exceeds 65ms before sending data. The other way to create a custom action which runs asynchronously so you have more of a fire-and-forget approach. This will run in a fraction of the time it takes when you are waiting for a response. If you don’t feel like creating your own I’ve done so already here so you can just install the solution.

All the code is on GitHub:
https://github.com/jlattimer/D365AppInsights

I’ve packaged the source files for distribution so that you can either use it as is or make your own modifications.

NuGet C# source package:
https://www.nuget.org/packages/JLattimer.D365AppInsights

Thursday, June 28, 2018

D365 Application Insights – JavaScript

If you’ve implemented the sample integration for Application Insights you know that a small JavaScript file was provided that was to be included on any forms where you wanted logging. This little script downloads a big script and then by having this, all sorts of useful data is to be magically logged. On the previously linked page it shows the user name, a page/entity name of some sort, and a page view duration measuring in the seconds. I’m not quite sure where that data is coming from but I ended up with data that looked more like this.

No way to identify which form the data came from, no way to identify which user (user_id doesn’t tie back to anything), and the page view durations are all in the milliseconds.

The browser timing (page load times) metrics weren’t any better.


495 millisecond page loads? That never happened… not ever.

It’s not all bad. It does capture exceptions on the form (not that you can trace it back to where), it provides some interesting data on browser use, client location, etc.

It also doesn’t tell you that there is an existing SDK that you can use to log your own telemetry. This is where what I did comes in. I’ve packaged everything together, but ultimately you still end up using the same small initialization script (that defaults some options to off) which pulls down the larger script but then I’ve added a layer on top that adds a few features and fixes some of the previously mentioned problems.

I’ve built onto these telemetry types:
  • Trace
  • Event
  • Metric
    • Page load duration (custom metric implementation)
    • Method execution duration (custom metric implementation)
  • Dependency
  • Exception
  • PageView
For examples of setting up and using, see the GitHub site’s wiki page.
By default with each request it’s going to log the following D365/CRM specific data as custom dimensions:
  • User Id
  • Entity Name
  • Entity Id
  • Form type
  • Form name
  • Organization name
  • Organization version
  • Source (JavaScript)
  • + any other custom dimensions of your choosing
This is in addition to the normal items that are logged like the date/time of the event, browser info, etc. It’s using the ids rather than friendly names to reduce overhead a bit. This could easily be updated to pull the friendly name or else you can cross reference it back to D365/CRM later. Either way, you’ll still have more contextual information about the data you’ve logged.

When setting up on a form you’ll pass a JSON configuration string as a parameter to the setup function. It controls what types of items will be logged (example turn off tracing in production), percentages of event types to log (so maybe you only 25% of total page view times), & debug mode. Exceptions at the form level should still be logged (regardless of you catching them or not), page view times, and page load times will be logged without any interaction so long as they aren’t disabled. All the other items rely on the developer to call the specific logging function when needed.

My implementation of a page view should be a bit more accurate as it logging the total time spent on the page just as the user is closing the form. And how does this happen? As I learned, all modern browsers have the Beacon API which includes the sendBeacon method whos exact purpose is to do things like send telemetry just as the user is leaving the page. Of course Internet Explorer 11 doesn’t have this and it’s still a supported browser, in which can I had to resort to a bit of a hack by including a very short synchronous request in the page’s beforeunload event. Not really perceptible but still ugly. 

Now my page view time is showing around 41 seconds, that sounds a little more reasonable. Also shown here is the current user id as user_AuthenticatedId and also the default D365/CRM specific custom dimensions.


Page load time isn’t really anything special (other than being more accurate) as it’s just using the browser’s performance APIs to read what the browser records as opposed to trying to start a timer yourself which would never really be close to accurate anyway. I’m logging them a custom metric.

image

7.4 seconds sounds a little more accurate.

Logging traces, metrics, events, caught exceptions, dependencies, and method execution times are all pretty straight forward. Logging XMLHTTPRequest dependency times an be done with 1 line of code. Pass the request and whatever name you want to the tracking function and then you won’t need to worry about wrapping a timer around anything or handling the end times in both successful and error events. Examples for everything is on the wiki page.

All the code is on GitHub:
https://github.com/jlattimer/D365AppInsights.Js

I’ve packaged the source files for distribution so that you can either use it as is or make your own modifications. If you’re going to being doing the later, read the notes here and/or look at the gulp file in the source code to see how the files are being put together.

NuGet JavaScript/TypeScript source package:
https://www.nuget.org/packages/JLattimer.D365AppInsights.Js

Monday, June 25, 2018

D365 Application Insights – Overview

There’s a good chance you’ve heard of Azure Application Insights, but if you’re in the CRM/D365 business there’s a good chance you stopped looking after not too long when you ran across either of these 2 things:
  1. The sample integration which was highlights between Application Insights and Dynamics 365 doesn’t provide a lot of value.
  2. Nearly all the .NET examples rely on an external assembly reference making it’s use in plug-in impossible.
If you were using it you were probably pushing your logging through an Azure Function or some other web service to take advantage of that pre-built assembly. But what they don’t advertise is that telemetry can be logged using simple HTTP requests. That bit of knowledge, some time spent examining the Application Insights JS code on GitHub, and some trial and error make it pretty clear how you can craft your own logging requests and take better advantage of the existing JS implementation to track some data that proves more useful than knowing how many users are still on Internet Explorer 7.

First though, why use Application Insights and not what’s already in D365? Really the only thing there is the plug-in trace log and this has its own shortcomings. No milliseconds, can’t create a record directly so you’d have to push anything from JavaScript through an action / plug-in, etc. Sure, we could model everything that Application Insights has in D365 as entities but then you’d be incurring all that extra overhead instead of offloading it elsewhere. Granted you would have the ability to use workflows for alerting and dashboards and other visualizations but is logging into the same system you are monitoring really a good idea?

Application Insights is a pretty capable logging platform. There are plenty of built in analytics and the ability to connect to Power BI to work with the data in just about any way you want, so long as you’ve captured some useful data. In our case that is data specific to D365. There are several pre-defined types of telemetry data that can be collected but all of them allow custom properties or dimensions as they are also known to be tracked as well. So that opens up the possibility of knowing which user, entity, form, etc. was involved in a log entry. The alerting capabilities give you more than just simply being able to send an email when there is an error but to be a little smarter about it by doing things like analyzing the number of failures over a period of time and only then notifying someone if a threshold is exceeded. Data is kept for 90 days so you can look at things like performance over time. If 90 days isn’t enough there is a continuous export feature to move the data to a different data store. It’s also cheap. $2.30 per month that includes 5GB of data, which is a lot of text. Those things are just scratching the surface, there is so much more beyond what I've described.

Types of telemetry

  • Dependencies – outbound calls to web services, SQL, or whatever
  • Exceptions – errors
  • Traces – messages typically for informational purposes
  • Events – application interactions
  • Metrics – numeric (single or aggregate) measurements
  • Page Views – duration a user stayed on a web page
  • +Others

Custom Dimensions

As I mentioned, additional data can be included along with any log entry in the form of key/value pairs. The values here can be strings, booleans, dates, or numbers.  So now when working with data inside the tools Application Insights provides, it can be queried or acted on just like any of the standard fields.

Custom Measurements

Much like custom dimensions, log entries can also include a set of measurements in the form of key/value pairs. The difference here is that the values must be numeric.

How To Start Logging

There isn’t any magic happening, you only need 3 things to get going.

Once you’ve got these things it’s just a matter of making the HTTP requests. It’s designed to be used from JavaScript so there are no CORS issues to contend with. No authentication to deal with, incoming telemetry is routed to the correct instance based on the Instrumentation Key.
I took what I learned and wrapped it up into a few different packages to facilitate logging:
  • D365AppInsights.Js – JavaScript & TypeScript libraries to log directly from front-end code
  • D365AppInsights – C# source code which can be added to a plug-in or custom workflow projects
  • D365Applnsights Actions – Actions with plug-ins that which can be used from just about anywhere
  • D365Applnsights Workflows – Custom steps to add logging to workflows

All the code is up on GitHub so I’m not going to post it here.

https://github.com/jlattimer/D365AppInsights.Js
https://github.com/jlattimer/D365AppInsights

In the next few posts I’ll dive a little deeper into each of the items.
If you’re in a hurry to use any of this or if I get busy and can’t get to the next post soon I think I’ve down at least a fair job of documenting things on the wiki pages on the respective GitHub sites.

NuGet C# source package
https://www.nuget.org/packages/JLattimer.D365AppInsights/
NuGet JavaScript/TypeScript source package
https://www.nuget.org/packages/JLattimer.D365AppInsights.Js/
Manages solutions for the prebuilt actions and workflow steps
https://github.com/jlattimer/D365AppInsights/releases

Monday, March 26, 2018

Updated CRM Workflow Utilities

I’ve just made a round up updates to my workflow utilities. I’ve added new functionality to most of them and did some refactoring on the backend. I’m also dropping support for the 2011/2013 version of the solutions I’d been keeping up mostly because they weren’t being downloaded very much and it frees up time for other endeavors. The current solution will work for anything 2015 and later (v7.0+).
If you’ve got suggestions for new functionality, make them on the corresponding GitHub issues page.


CRM Email Workflow Utilities

GitHub    Latest Release
  • Email Business Unit
  • CC Business Unit
  • Email Security Role
  • CC Security Role
  • Email Team
  • CC Team
  • Email Queue Members
  • CC Queue Members
  • Email Connection - NEW
  • CC Connection - NEW
  • Delete Email Attachments
  • Delete Email Attachments By Name
  • Send Draft Email
  • Check Email Attachments Present & Count
  • Create Template – NEW

CRM DateTime Workflow Utilities

GitHub    Latest Release
  • Add Business Days
  • Add Days
  • Add Hours
  • Add Minutes
  • Add Months
  • Add Years
  • DateDiff - NEW
  • Date Difference Seconds - NEW
  • Date Difference Hours
  • Date Difference Minutes
  • Date Difference Days
  • Date Difference Months
  • Date Difference Years
  • Get Second Number - NEW
  • Get Minute Number - NEW
  • Get Hour Number - NEW
  • Get Day Number
  • Get Month Number
  • Get Year Number
  • Get Month Name
  • Get Day Number Of Week
  • Get Day Number Of Year
  • Get Day Of Week
  • Get Quarter Number Of Year
  • Get Week Number Of Year
  • Get Month Start/End Dates
  • Get Quarter Start/End Dates
  • Get Week Start/End Dates
  • Get Year Start/End Dates
  • Is Between
  • Is Business Day
  • Is Same Day
  • Relative Time String
  • ToDateTime
  • To UTC String
  • Set Time - NEW
  • Get Formatted Date String - NEW
  • Set DatePart – NEW

CRM String Workflow Utilities

GitHub    Latest Release
  • B64 Decode - NEW
  • B64 Encode - NEW
  • Contains
  • Create Empty Spaces
  • Decode Html - NEW
  • Encode Html - NEW
  • EndsWith
  • Join
  • Length
  • PadLeft
  • PadLeft Dynamic
  • PadRight
  • PadRight Dynamic
  • Random String - NEW
  • Regex Extract
  • Regex Match
  • Regex Replace
  • Regex Replace With Space
  • Replace
  • Replace With Space
  • Reverse
  • StartsWith
  • Substring
  • ToLower
  • ToTitleCase
  • ToUpper
  • Trim
  • Word Count
  • Remove HTML - NEW
  • Url Encode - NEW
  • Url Decode- NEW

CRM Numeric Workflow Utilities

GitHub    Latest Release
  • Add
  • Average
  • Divide
  • Max
  • Min
  • Multiply
  • Random Number
  • Random Number Between
  • Round
  • Subtract
  • ToDecimal
  • ToInteger
  • Truncate
  • Absolute Value - NEW
  • IsStringNumeric – NEW

CRM Note Workflow Utilities

GitHub    Latest Release
  • Check Attachment
  • Copy Note
  • Move Note
  • Delete Attachment
  • Delete Attachment By Name
  • Delete Note
  • Update Note Text
  • Update Note Title

Monday, March 12, 2018

Log to Application Insights from Microsoft Flow or Logic Apps

If you haven’t checked out Azure Application Insights yet you might want to give it a look. It’s got a lot to offer in terms of  logging, monitoring, alerting, multi-colored charts and graphs, etc. Microsoft provides libraries for several languages to make logging things easier but ultimately you’re just making HTTP requests to a public endpoint.

There isn't a connector which lets you write to Application Insights but logging can be done fairly easily with a HTTP Action. I’d suggest referencing the telemetry API to figure out what options are available and how data needs to be structured as my examples won’t contain everything available.

Overview

All the requests will be:

Method: POST
Uri: https://dc.services.visualstudio.com/v2/track

In each request body:
  • Replace “time” with the Flow utcNow() function
  • Replace "00000000-0000-0000-0000-000000000000" with your Application Insights key
  • Replace “properties” with any custom key/value pairs you wish to track

Trace

Diagnostic log messages

Body:
  • Replace “message” with whatever you want to trace

Event

User actions and other events

Body:
  • Replace “name” with the name of the event
  • Replace inside “measurements” with a measurement name (string) and value (numeric) or set to null if not using

Metric

Performance measurements such as queue lengths not related to specific events

Body:
  • Replace inside “metrics” with a metric name (string), kind (integer), value (double) – see API for additional details

Dependency

Logging the duration and frequency of calls to external components that your app depends on

Body:

Recommended replacements, doesn’t need to follow this format exactly
  • Replace “id” with an id of some sort - Application Insights libraries have a way of generating an id if you care to track it down
  • Replace “name” with the HTTP verb and Url Absolute Path
  • Replace “resultCode” with the HTTP Status Code
  • Replace “duration” with the elapsed time
  • Replace “success” with true/false
  • Replace “data” with the HTTP verb and Url
  • Replace “target” with Url Host
  • Replace “type” with something describing the type of request
Tracking the time it took to execute a different HTTP request within a Flow was a little difficult. Catching the request Start and End times was fairly easy but I couldn’t seem to find a way to easily calculate the duration. I cheated a little and created an Azure Function to do it for me – not exactly ideal but whatever. Once I had that I could plug the “Calculate Duration” HTTP Action Body into the “duration”. The “resultCode” and be pulled directly from the HTTP Action. To populate “success” I used a simple formula which looked at the Status Code and set accordingly.

if(equals(outputs('Test_HTTP')['statusCode'], 200),true, false)

image

The Azure Function to calculate duration looks like this:

And that’s about it, you should start seeing results in your Application Insights instance.

https://github.com/jlattimer/MSFlowAppInsights