Monday, November 12, 2018

Use Xrm.WebApi in a stand alone Web Resource

I ran into this issue working on the last update to CRM REST Builder when adding the Xrm.WebApi functionality. Note this won’t effect you if you’ve got a web resource embedded in a form.

Attempting to execute a simple retrieve for example:"account"…
And I was greeted with this error:

Unexpected token u in JSON at position 0

What does that even mean? I bit of searching and I ran across this Stack Overflow question. So it seems Xrm.WebApi is looking for a variable which maps entity names with their corresponding entityset names. Easy fix since I had all the data so I dynamically built a variable with all the mappings which looked a bit like this (you only need to add the entities you're working with):
window["ENTITY_SET_NAMES"] = ['{"account":"accounts", "contact":"contacts"}'];
Got further along and started working implementing actions and functions and ran into the same error once again, WTF. After some tedious debugging through the Microsoft code I found there is another dependency on a similar variable. This one maps entities to their primary id fields. So I added something like this with all these mappings:
window["ENTITY_PRIMARY_KEYS"] = ['{"account":"accountid", "contact":"contactid"}'];
After that my problems were solved.

Monday, October 15, 2018

CRM REST Builder v2.6.0.0

So here it is… finally.
  • Added 9.x support
  • Added Xrm.WebApi
  • Better support for Action/Function parameters
    • Collection handling
    • Entity related parameters allow choosing the entity type if not specific and determines id field
  • Fixed a bunch of bugs
    • Displaying XML in results
    • Issue with fields containing “_value”
    • Decimal parameter conversion
    • Function parameter passing via query string
  • Other usability enhancements
    • Default unpopulated ids to empty guid when generating code
Also no longer supporting the 2011/2013 versions. They are still available on GitHub, just no new updates.

This will probably be the last major update for this so I can focus on creating a Web API only version that runs in the XrmToolBox.

Download here for CRM 2015/2016 & D365 CE (7.0+)

Log any issues here:

Thursday, July 5, 2018

D365 Application Insights – Plug-ins & Custom Workflows

Not too many people were probably using Application Insights directly in plug-ins and custom workflow steps since it was introduced because every example I ever saw was writing using set a set of assemblies to making interfacing with the service easy (much like we do with D365). Of course awhile back I tried to ILMerge those assemblies into a plug-in but needless to say that didn’t work. I know a few instances where individuals had the logging proxied through something like an Azure Function which I’m sure worked perfectly well but then also took on the extra dependency (and cost) of whatever was doing the relaying. Luckily now we can avoid doing all that because we’ve now established there is a REST API and we can call it from just about anywhere.

I put a wrapper around it using C# so that you can easily include it plug-ins and workflow code. It’s ready to use right away but of course it can be modified as you see fit.
I’ve built in these telemetry types:
  • Trace
  • Event
  • Metric
  • Dependency
  • Exception
For examples of setting up and using, see the GitHub site’s wiki page.
By default with each request it’s going to log the following D365/CRM specific data as custom dimensions:
  • ExecutionContext
    • InitiatingUserId
    • ImpersonatingUserId
    • CorrelationId
    • Message
    • Mode
    • Depth
    • InputParameters
    • OutputParameters
    • Stage (plug-in)
    • WorkflowCategory (workflow)
    • PrimaryEntityName
    • PrimaryEntityId
    • OrganizationName
  • Organization version
  • Source (Plug-in or Workflow)
This is in addition to the normal items that are logged like the date/time of the event, originating location (datacenter city & state), etc.. Again it’s using the ids rather than friendly names to reduce overhead a bit. This could easily be updated to pull the friendly name or else you can cross reference it back to D365/CRM later.

Logging traces, metrics, events, caught exceptions, and dependencies are all pretty straight forward. Examples for everything is on the wiki page.

The constructor of the logger will need a configuration passed to it in terms of an object or a JSON string so you’ll need to have a way to get that information into your code by using the plug-ins configurations, workflow inputs, etc. You can control the types of telemetry logged, percentages of the events to log, or to enable and disable context tracking.

Besides context values if you so choose, nothing logged will be automatic. You’ll need to incorporate the Application Insights logger into your own exception handling and performance tracking.

The other important point to bring up is the actual duration it takes to log something. On average from D365 Online it was taking ~700ms to get a response back that the item was accepted. Doesn’t sound like much but if your plug-in is already busy, this isn’t going to help. The first way to mitigate that is to only log a percentage of “normal”, non-exception events. This was one of the driving factors to build in the functionality right away. That operates at a broad level across the logging instance and telemetry type. You can always add it your own filters to reduce the number of normal results. For example if you are logging the duration it takes to call an external web service and it normally take 50ms to complete, wrap the logging statement in something that checks to see if it exceeds 65ms before sending data. The other way to create a custom action which runs asynchronously so you have more of a fire-and-forget approach. This will run in a fraction of the time it takes when you are waiting for a response. If you don’t feel like creating your own I’ve done so already here so you can just install the solution.

All the code is on GitHub:

I’ve packaged the source files for distribution so that you can either use it as is or make your own modifications.

NuGet C# source package:

Thursday, June 28, 2018

D365 Application Insights – JavaScript

If you’ve implemented the sample integration for Application Insights you know that a small JavaScript file was provided that was to be included on any forms where you wanted logging. This little script downloads a big script and then by having this, all sorts of useful data is to be magically logged. On the previously linked page it shows the user name, a page/entity name of some sort, and a page view duration measuring in the seconds. I’m not quite sure where that data is coming from but I ended up with data that looked more like this.

No way to identify which form the data came from, no way to identify which user (user_id doesn’t tie back to anything), and the page view durations are all in the milliseconds.

The browser timing (page load times) metrics weren’t any better.

495 millisecond page loads? That never happened… not ever.

It’s not all bad. It does capture exceptions on the form (not that you can trace it back to where), it provides some interesting data on browser use, client location, etc.

It also doesn’t tell you that there is an existing SDK that you can use to log your own telemetry. This is where what I did comes in. I’ve packaged everything together, but ultimately you still end up using the same small initialization script (that defaults some options to off) which pulls down the larger script but then I’ve added a layer on top that adds a few features and fixes some of the previously mentioned problems.

I’ve built onto these telemetry types:
  • Trace
  • Event
  • Metric
    • Page load duration (custom metric implementation)
    • Method execution duration (custom metric implementation)
  • Dependency
  • Exception
  • PageView
For examples of setting up and using, see the GitHub site’s wiki page.
By default with each request it’s going to log the following D365/CRM specific data as custom dimensions:
  • User Id
  • Entity Name
  • Entity Id
  • Form type
  • Form name
  • Organization name
  • Organization version
  • Source (JavaScript)
  • + any other custom dimensions of your choosing
This is in addition to the normal items that are logged like the date/time of the event, browser info, etc. It’s using the ids rather than friendly names to reduce overhead a bit. This could easily be updated to pull the friendly name or else you can cross reference it back to D365/CRM later. Either way, you’ll still have more contextual information about the data you’ve logged.

When setting up on a form you’ll pass a JSON configuration string as a parameter to the setup function. It controls what types of items will be logged (example turn off tracing in production), percentages of event types to log (so maybe you only 25% of total page view times), & debug mode. Exceptions at the form level should still be logged (regardless of you catching them or not), page view times, and page load times will be logged without any interaction so long as they aren’t disabled. All the other items rely on the developer to call the specific logging function when needed.

My implementation of a page view should be a bit more accurate as it logging the total time spent on the page just as the user is closing the form. And how does this happen? As I learned, all modern browsers have the Beacon API which includes the sendBeacon method whos exact purpose is to do things like send telemetry just as the user is leaving the page. Of course Internet Explorer 11 doesn’t have this and it’s still a supported browser, in which can I had to resort to a bit of a hack by including a very short synchronous request in the page’s beforeunload event. Not really perceptible but still ugly. 

Now my page view time is showing around 41 seconds, that sounds a little more reasonable. Also shown here is the current user id as user_AuthenticatedId and also the default D365/CRM specific custom dimensions.

Page load time isn’t really anything special (other than being more accurate) as it’s just using the browser’s performance APIs to read what the browser records as opposed to trying to start a timer yourself which would never really be close to accurate anyway. I’m logging them a custom metric.


7.4 seconds sounds a little more accurate.

Logging traces, metrics, events, caught exceptions, dependencies, and method execution times are all pretty straight forward. Logging XMLHTTPRequest dependency times an be done with 1 line of code. Pass the request and whatever name you want to the tracking function and then you won’t need to worry about wrapping a timer around anything or handling the end times in both successful and error events. Examples for everything is on the wiki page.

All the code is on GitHub:

I’ve packaged the source files for distribution so that you can either use it as is or make your own modifications. If you’re going to being doing the later, read the notes here and/or look at the gulp file in the source code to see how the files are being put together.

NuGet JavaScript/TypeScript source package:

Monday, June 25, 2018

D365 Application Insights – Overview

There’s a good chance you’ve heard of Azure Application Insights, but if you’re in the CRM/D365 business there’s a good chance you stopped looking after not too long when you ran across either of these 2 things:
  1. The sample integration which was highlights between Application Insights and Dynamics 365 doesn’t provide a lot of value.
  2. Nearly all the .NET examples rely on an external assembly reference making it’s use in plug-in impossible.
If you were using it you were probably pushing your logging through an Azure Function or some other web service to take advantage of that pre-built assembly. But what they don’t advertise is that telemetry can be logged using simple HTTP requests. That bit of knowledge, some time spent examining the Application Insights JS code on GitHub, and some trial and error make it pretty clear how you can craft your own logging requests and take better advantage of the existing JS implementation to track some data that proves more useful than knowing how many users are still on Internet Explorer 7.

First though, why use Application Insights and not what’s already in D365? Really the only thing there is the plug-in trace log and this has its own shortcomings. No milliseconds, can’t create a record directly so you’d have to push anything from JavaScript through an action / plug-in, etc. Sure, we could model everything that Application Insights has in D365 as entities but then you’d be incurring all that extra overhead instead of offloading it elsewhere. Granted you would have the ability to use workflows for alerting and dashboards and other visualizations but is logging into the same system you are monitoring really a good idea?

Application Insights is a pretty capable logging platform. There are plenty of built in analytics and the ability to connect to Power BI to work with the data in just about any way you want, so long as you’ve captured some useful data. In our case that is data specific to D365. There are several pre-defined types of telemetry data that can be collected but all of them allow custom properties or dimensions as they are also known to be tracked as well. So that opens up the possibility of knowing which user, entity, form, etc. was involved in a log entry. The alerting capabilities give you more than just simply being able to send an email when there is an error but to be a little smarter about it by doing things like analyzing the number of failures over a period of time and only then notifying someone if a threshold is exceeded. Data is kept for 90 days so you can look at things like performance over time. If 90 days isn’t enough there is a continuous export feature to move the data to a different data store. It’s also cheap. $2.30 per month that includes 5GB of data, which is a lot of text. Those things are just scratching the surface, there is so much more beyond what I've described.

Types of telemetry

  • Dependencies – outbound calls to web services, SQL, or whatever
  • Exceptions – errors
  • Traces – messages typically for informational purposes
  • Events – application interactions
  • Metrics – numeric (single or aggregate) measurements
  • Page Views – duration a user stayed on a web page
  • +Others

Custom Dimensions

As I mentioned, additional data can be included along with any log entry in the form of key/value pairs. The values here can be strings, booleans, dates, or numbers.  So now when working with data inside the tools Application Insights provides, it can be queried or acted on just like any of the standard fields.

Custom Measurements

Much like custom dimensions, log entries can also include a set of measurements in the form of key/value pairs. The difference here is that the values must be numeric.

How To Start Logging

There isn’t any magic happening, you only need 3 things to get going.

Once you’ve got these things it’s just a matter of making the HTTP requests. It’s designed to be used from JavaScript so there are no CORS issues to contend with. No authentication to deal with, incoming telemetry is routed to the correct instance based on the Instrumentation Key.
I took what I learned and wrapped it up into a few different packages to facilitate logging:
  • D365AppInsights.Js – JavaScript & TypeScript libraries to log directly from front-end code
  • D365AppInsights – C# source code which can be added to a plug-in or custom workflow projects
  • D365Applnsights Actions – Actions with plug-ins that which can be used from just about anywhere
  • D365Applnsights Workflows – Custom steps to add logging to workflows

All the code is up on GitHub so I’m not going to post it here.

In the next few posts I’ll dive a little deeper into each of the items.
If you’re in a hurry to use any of this or if I get busy and can’t get to the next post soon I think I’ve down at least a fair job of documenting things on the wiki pages on the respective GitHub sites.

NuGet C# source package
NuGet JavaScript/TypeScript source package
Manages solutions for the prebuilt actions and workflow steps