Creating a Coveo facet with Sitecore sub-domains and external sites as options

Take this example: you have multiple sub-domains hosted in your Sitecore site, and each one has its own node under Content and its own Home item. You also have various external sites not hosted in Sitecore as completely separate sites. You’re building a search page with Coveo and want to develop a basic facet to list all of your internal sub-domains and external sites so that the end-users can filter pages and content by site. How would one do this? In this blog post, I will explain. Disclaimer: When I did this, I was working with Coveo for Sitecore v4.1 – Hive Framework, so if your version is much different, it’s possible not everything in this post will be accurate. Additionally, please note that the XML/HTML code snippets in this blog post will have the greater than/less than symbols stripped out because WordPress can’t handle those.

Create a Coveo Field for Your Facet

Every new facet you create will need its own field, so lets create one. This is going to be a string, single-value field representing the name of a particular site (i.e. “Store” or “Marketing”, etc). In your Coveo.SearchProvider.Custom.config file, add a new field entry in the fieldMap/fieldNames node. In my case, my field’s name was aopaSiteName, so that naming convention will be used in other parts of this post as well. Your field entry should look something like this:

fieldType fieldName="aopaSiteName" settingType="Coveo.Framework.Configuration.FieldConfiguration, Coveo.Framework" isExternal="true" isFacet="true" /

Note that the field definition has isExternal="true" – this specifies that the field is not a normal Coveo for Sitecore field, so the field name won’t need to be converted. So if you put isExternal="false", your field name will become something like fmyfield99999 instead of just myfield. I suggest you keep this setting true to make development easier. Then, isFacet=true is needed so that this field can be used in a facet.

Creating a Computed Field

First, let’s focus on your internal content. What we need to do is actually rather simple: first, set up a basic computed field. See this page for basic information on how to do that with Coveo, and this page for a code sample if you’re not sure how the code for this would look. Inside the ComputeFieldValue method (see the second link above), you’ll want to get the indexable item, then the actual Sitecore item, and perform null checks on both so you have something you can work with:

        public object ComputeFieldValue(IIndexable indexable)
        {
            var indexableItem = indexable as SitecoreIndexableItem;
            if (indexableItem == null) return "Unknown";

            var item = indexableItem.Item;
            if (item == null) return "Unknown";

            ...
        }

Then, you just need to return a string value for each sub-domain node under Content. You can do this in multiple ways; using Glass is advised if possible. I decided to simply check the path and return the appropriate string for that path:

            ...

            if (item.Paths.Path.Contains("Sites/AOPA Main") ||
                item.Paths.Path.Contains("sitecore/media library")) return "AOPA";
            if (item.Paths.Path.Contains("Sites/Insurance")) return "Insurance";
            if (item.Paths.Path.Contains("Sites/Finance")) return "Finance";
            if (item.Paths.Path.Contains("Sites/PPS")) return "PPS";
            if (item.Paths.Path.Contains("Sites/You Can Fly")) return "You Can Fly";
            if (item.Paths.Path.Contains("Sites/Foundation")) return "Foundation";

            return "AOPA";
        }

Next, you’ll need to add a computed index field reference in your Coveo.SearchProvider.Custom.config file, under fields hint="raw:AddComputedIndexField", like this:

field fieldName="AOPASiteName" sourceField="AOPA.Library.Search.CoveoFields.SiteName, AOPA.Library">AOPA.Library.Search.CoveoFields.SiteName, AOPA.Library

Now, rebuild the Coveo indexes so that new field gets added to the index.

Adding Mappings on External Sources

In this example, since we’re building a facet with options showing each internal and external site, you’ll want to add a field mapping for each external site that you want in that list. You can add one by selecting a source and clicking the three dots, then click Manage mappings… Add a new mapping and select the field you created earlier. Then for the rule, simply put the name of the current site in plain text (i.e. “Blog”). Refer to online documentation for how to create a mapping in more depth. Once done, your mapping should look something like this:

aopa site name mapping

Now, add the field as a mapping for all your other external sites, then (preferably once you’ve added all mappings necessary for other features), re-build the source for your external site.

Reference Your External Sources

We need to find a way to reference your external sources in the search interface. Again, I did this with Coveo Hive, so if you are using the legacy framework, what you need to do will be different (and unfortunately I can’t help you there). First off, if you haven’t already created a custom version of your Search Interface view (usually located at /Views/Coveo Hive/Search Interfaces), do that now and place it under /Views/Coveo Hive/Custom/Search Interfaces and create any new directories necessary.

Then, in your Search Interface view, in the main div which iterates through the raw properties, add a new div with class CoveoForSitecoreExternalContent and a data attribute of data-sc-sources, and make the value of that attribute a comma delimited string of the names of your external sources. For example, here’s mine:

div class="CoveoForSitecoreExternalContent" data-sc-sources='AOPA Airports Directory,AOPA Blog,AOPA Brightcove Videos,AOPA Hangar'>
The names referenced there need to be exact as they are in the cloud platform. As of the writing of this post, source names in Coveo Cloud V2 can’t be changed after the source is created, so no worries regarding a source getting renamed and then you having to update it in the view. Here is where you might want to put a little extra effort in, perhaps find a way to render the value of a Sitecore field here so you can manage this string from Sitecore instead.

Completion

Finally, add a new facet to your search interface using the Experience Editor and select the field you created (or type in the field name). Once the facet renders it should show you options for all the sites you referenced in your code or in the source mappings.

sites facet completion

Advertisements

The search query is denied because it originated from a Coveo quick view

A few months ago I was troubleshooting an issue with a failing Coveo index rebuild. Several times, while reviewing the Sitecore logs, I noticed an error occurring very frequently (~250k times in one log file) after the index rebuild process had started:

WARN  The search query is denied because it originated from a Coveo quick view. For more information, enable DEBUG logging for the 'Coveo.SearchProvider.LinqToCoveoIndex`1[TItem]' class.

I put this error message into Google only to find absolutely nothing. Our search page wasn’t even using quick views, so why was this error showing up in the logs? I enabled DEBUG logging but was not able to find a cause of the error. I contacted Coveo Support.

Wait… What is a Quick View Anyway?

Basically it’s a feature of Coveo that lets you preview a page in the index, quickly. You can read more about it here. Even if you aren’t using quick views on your search page, Coveo still uses and renders them. For example, if you open an indexed item’s properties in the Content Browser in Coveo Cloud, you will see a quick view tab. Upon opening that tab, Coveo generates a preview of your page. It’s much like opening the page manually in a browser, except the page renders much faster in a quick view.

Understanding the Issue

When a Coveo index is rebuilding, Coveo visits the item to fill its quick view. If Coveo is visiting a page where you have a very complex component or module running, that might prompt this error to occur. For example, in my client’s site they have a component named Related Articles, which basically shows a list of other articles that are related to this one. This site happened to have hundreds of thousands of articles, all which needed to be indexed, so the Coveo crawler was hitting these articles and their related articles over and over again.

Solutions

Coveo Support provided an article with a potential solution. Apparently, adding a Request.UserAgent check in complex components on indexed pages is one option. This can be placed on a view, model or sublayout, as the article states, but I also found out that you can add it to your back end code, too; and in my case, that seemed like the best thing to do. In the code that gets the related articles, I put this in at the very beginning of the method:

    if (HttpContext.Current != null && HttpContext.Current.Request != null && HttpContext.Current.Request
        .UserAgent.Contains("Coveo Sitecore Search Provider"))
    {
        return null;
    }

After rebuilding the indexes, the quick view warning was suddenly gone.

Another option Coveo provided was to disable queries from quick views completely. You could add the code in Step 1 of this article to your custom search provider config, but supply a value of false. As the document states,

This setting acts as a master switch. If the setting is set to false, the search queries are never executed in the context of the quick view.

Conclusion

The point is, if you have any complex logic running on a page you know you are indexing, you need to handle that via one of the two above solutions. This could speed up your rebuild time too.

#dotNETConf, Day 1 – Summary

Day 1 is in the books! I’ll admit I didn’t see everything, only certain sessions, but I thought I’d share what I got out of the free, virtual .NET conference hosted by Microsoft on Channel 9.

Keynote – Scott Hunter

I loved the Keynote; they got right to the heart of what’s new and cool with .NET, specifically explaining what .NET Standard is, how .NET Core is evolving and such. They went over the new templating engine – how you can use dotnet new in a console to create a new project without the need for Visual Studio. That command will show a list of available projects you can create. You would then run dotnet new Console, for instance, to create a new C# Console application.

One really cool thing they showed was the ability to edit a .csproj file without having to unload the project. I am LOVING these quality of life improvements Microsoft has been rolling out! In fact, last week I learned that in Visual Studio 2015 you can evaluate lambda expressions in a Watch window, which blew my mind, so I can’t wait to start using the newer versions of this great IDE.

Scott then showed Razor Pages, which as Razor views with .cshtml.cs code behind files included. Pretty cool, seems like MVC for Slackers basically. Probably not for me personally. Then, he showed Application Insights and Snapshots in Azure, which can provide detailed information about errors the application ran into. Good stuff.

Mobile .NET – Miguel de Icaza

Miguel showed a really cool feature of Visual Studio for Mac, where you can specify code to be run live. Meaning any change you make to the code will automatically propagate to the mobile phone / front end without having to save the file. He also shared something called the Embedinator-4000, which basically turns any .NET library into a native library, and it supports a range of things:

embedinator4000.png

Visual Studio 2017 & C# 7.x – Mads Torgersen

New notable features are available in VS 2017, including:

  1. Code Styles – setting code standards in the IDE. VS will tell you when you deviate from the standards you set up.
  2. Live Unit Testing – as you are writing a class, or a controller, you can have a unit test running and providing pass/fail information as you type
  3. Smudges – small dots under a class name that might indicate suggestions – one example:

smudge.png

.NET Foundation – Jon Galloway

This session explained what .NET Standard is and how they are improving it. I thought .NET Standard meant .NET Framework before I attended this session. .NET Standard is a set of APIs that all .NET platforms have to implement (including .NET Framework). It’s a specification, not a thing you install, and it’s primary use is code reuse across all platforms that use it. The higher your .NET Standard version, the more APIs you get; the lower the version, the more platforms support it. Jon demoed a web application that shows which APIs are available in which .NET Standard versions; I don’t think it’s live anywhere, but looks pretty useful.

Diagnostics 101 – Jon Skeet

The purpose of this task was to inform people how to diagnose and troubleshoot programming issues better, before posting on Stack Overflow. That’s something I was guilty of when I first starting writing code so I can relate. The ones I got the most out of:

  • Divide and Conquer – Reduce the size of the codebase you’re looking at when trying to solve a problem. Essentially, he’s saying that you need to do some trial and error, eliminate some pieces of code that you think may be where the bug is hiding, and keep doing that until you uncover the bug. It’s common sense to me.
  • Challenge Assumptions – Your code is probably wrong. Question everything. This is basically what happens to me every time I start writing a Stack Overflow question. There are so many questions I started that I didn’t end up submitting because I realized what the problem was while I was typing it into the site, it’s incredible.
    • Things to be skeptical of:
      • Your code
      • The debugger’s immediate window – if it seems off, don’t trust it
      • In Watch, beware of the debugger’s string repesentation

Summary

Overall, really good first day. Not sure how much I’ll be tuning in for Day 2, unless I have absolutely nothing else to do.

#MKEDOTNET 2017 was a blast!

This past Saturday I attended MKE DOT NET for the third time. MKE DOT NET is a .NET developer conference based in the Milwaukee area. The previous two years the conference was held in a hotel in Pewaukee. This year, they changed things up a bit and hosted it at Potowatomi Casino near downtown Milwaukee. Getting there was a bit difficult because there were a lot of closed off streets, but once I got there it was really nice.

Keynote Speaker: Maria Naggaga

Lots of people are interested in developing in .NET, but the main complaint was that it’s too difficult to learn. Maria presented some up and coming options to make .NET easier to learn for people who are currently developing in other frameworks and languages, like Python or Ruby for example.

First, she listed off some recommended code school sites. This was helpful for me because I’ve been trying to get my brother into coding with .NET and C#. Sites included Free Code Camp, Code School, Treehouse, and Javascript.com. Next, she showed an upcoming version of docs.microsoft.com, which will include everything in the current Docs site, but added a Run button for every code snippet. That part wowed me; I know one of the big annoyances with developers learning .NET was having to install Visual Studio to do any coding, so this goes around that. Great job MSFT!

After the keynote, I attended several sessions. I will list out the sessions which I got the most out of below:

Hardening Your ASP.NET Core Applications – Brice Williams and Robert Hansen

This session went over securing ASP.NET Core applications. Brice and Bob showed how to enforce HTTPS on a site at the global level or controller level. Additionally, they went through some common security risks, such as Cross Site Scripting (XSS) and Cross Site Request Forgery ( CSRF), and how to counteract those in your application. One cool thing I learned was that Javascript UI frameworks like React or Angular will do the HTML encoding for you as a way to protect against CSRF. The most important point made was to not create a security system yourself; it is extremely difficult and there’s probably a system available that someone else made that would work for you.

Cool SQL Server Features Every Developer Should Know About – David Berry

This presentation went over some new features in SQL Server 2016:

  1. Temporal tables – Every time a row is changed, SSMS keeps a snapshot of the old row values. This allows us to see what the value in a column was at any given date and time. The history table is created and maintained automatically by SQL Server, removing the need for us to write triggers.
  2. JSON Support – Allows you to store JSON data in a VARCHAR() data type. There are some new functions that go along with this, such as JSON_VALUE to get a value from a JSON object,  and JSON_QUERY to pull out an entire array or object.

David also showed Windowing Functions, which was actually released with SQL Server 2008 and later. “Group By on steroids”, Windowing Functions can partition and aggregate data by different criteria in the same statement. One example was the RANK() function, which can find the highest value, partitioning over various columns, and provide a rank for each row. I really need to look into these functions more…

Building Reusable UI Components in ASP.NET Core MVC – Scott Addie

.vbhtml files exist. It’s true. That wasn’t the point of this talk though. Scott showed us a brand new feature in ASP.NET Core V1 called Tag Helpers. Using this feature you can express server-side processing via HTML elements. An example would be:

<hand player="John">

Hand is the class, Player is a property of Hand. Tag Helpers are really cool for a few reasons:

  1. They offer great IntelliSense support.
  2. They eliminate context switching between HTML and C# that you would get with Razor.
  3. They were designed with testing in mind.

One other cool thing I learned is the C# nameof() feature, which can  be used to avoid hard coding class name strings. Usage: i.e. nameof(MyClass) would output “MyClass” as a string.

Conclusion

This event keeps getting better every year. Particlarly, the SQL Server Features and UI Components sessions amazed me! I learned a lot of cool things in this conference, and made note of a lot of things to look into so that I can learn more. Thanks Centare for hosting another great MKE DOT NET!

Creating a dynamic image in Rich Text fields

Have you ever inserted an image, then changed its alt or title tags in the Media Library and noticed that the change does not propagate into the rich text field? It’s because Sitecore only inserts static values into the field HTML when you insert an image. If you’ve ever wanted or needed those images to act dynamically – so that changing alt or title text affects the field HTML -then this article might help you.

The Secret? Tokens!

We can modify the rich text editor dialogs so that when a user inserts Sitecore media, the ALT and TITLE text being sent is actually just a token – i.e. {{alt}}. There is no specific syntax required, just use something that you like. That means that we’re actually inserting “{{alt}}” and “{{title}}” into the field HTML, not the actual (static) values. The goal then is to create a pipeline processor that will read the field and look for those tokens, and replace them with the current values of the Alt and Title fields of the media item.

Customizing the InsertImage dialog

The first thing we will want to do is modify the InsertImage dialog. The InsertImage dialog is a Sitecore dialog, and the source can be found with any Sitecore installation. You would need two files:

  1. The InsertImage.xml file for the dialog, which can be found in the webroot of your site under sitecore\shell\Controls\Rich Text Editor/InsertImage.
  2. The InsertImageForm.cs file (code behind file) which can be extracted from the Sitecore.Client.dll. You can use .NET Reflector, DotPeek, or JustDecompile (I personally use the last one). Look for a Sitecore.Shell.Controls.RichTextEditor.InsertImage entry.

You will want to take these two files and preferably put them in your solution. Place these in a new folder, e.g. Controls/RichTextEditor/InsertSitecoreMediaExtended and make the following changes:

  • Keep the .xml file name the same. In the .xml file, there is a line which points to the code behind file – update that with the location of the file in the solution and the assembly name.
  • Update the name of the .cs file to match the parent folder (InsertSitecoreMediaExtended.cs). In that file, remove everything except the OnOK and SetDimensions methods. You will need to modify the OnOK method so that alt and title are set using tokens:
image.Add("Alt", StringUtil.EscapeQuote("{{alt}}"));
image.Add("Title", StringUtil.EscapeQuote("{{title}}"));

Optional: You can edit the Telerik.Web.UI.Editor.DialogControls.SetImageProperties dialog by adding a title field and checkboxes next to the alt and title fields to make it easier for content editors to specify this option.

Let’s make a new pipeline!

You will want to extend the RenderField pipeline in Sitecore.Kernel, specifically the GetFieldValue processor. So when Sitecore gets the value of the RTE field, we’re going to modify the HTML by swapping tokens for media item values. Create a new folder in your pipelines folder or project named GetFieldValueExtended, and a new class file under it with the same name.

Now you’ll need to tell Sitecore to run your custom processor. There are a few options – patch:before, patch:after or overwrite completely. I personally used the patch:after option. If you choose to overwrite, make sure your processor implements all of the same code as the original. Create a new patch file in the App_Config/Include folder similar to the below. (In my case, the solution has a Processor project.)

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <pipelines>
            <renderField>
                <processor
                    type="MyProject.Processor.RenderField.GetFieldValueExtended, MyProject.Processor"
                    patch:after="processor[@type='Sitecore.Pipelines.RenderField.GetFieldValue, Sitecore.Kernel']" />
            </renderField>
        </pipelines>
    </sitecore>
</configuration>

In your custom processor class file, set up the Process event with some Asserts and other logic needed before doing the swap. Create the tokens as constants for flexibility. Here is the full code used, which is currently running in a production environment with no issues. I highly recommend HtmlAgilityPack for spelunking in the HTML. (NOTE: The code assumes your production environment Sitecore database is web, which may not be true in your case.)

    public class GetFieldValueExtended
    {
        private const string AltToken = "{{alt}}";
        private const string TitleToken = "{{title}}";

        public void Process(RenderFieldArgs args)
        {
            Assert.ArgumentNotNull(args, "args");
            Assert.ArgumentNotNull(args.Item, "args.Item");
            Assert.ArgumentNotNull(args.Item.Database, "args.Item.Database");
            Assert.ArgumentNotNull(args.GetField(), "args.GetField()");

            if (args.Item.Database.Name.Equals("web"))
            {
                if (args.GetField().Value.Contains(AltToken) || args.GetField().Value.Contains(TitleToken))
                {
                    args.Result.FirstPart =  ReplaceTokens(args.GetField().Value);
                }
            }
        }

        private string ReplaceTokens(string fieldValue)
        {
            if (fieldValue.Contains("img") == false) return fieldValue;

            var document = new HtmlAgilityPack.HtmlDocument();
            document.LoadHtml(fieldValue);

            foreach (var img in document.DocumentNode.Descendants("img"))
            {
                var imgSrc = img.Attributes["src"];
                if (imgSrc == null) continue;

                // Remove query string from src (if it exists)
                var mediaUrl = CleanImageSource(imgSrc.Value);

                // Get media item ID based on media url
                DynamicLink dynamicLink;
                if (!DynamicLink.TryParse(mediaUrl, out dynamicLink)) continue;
                MediaItem mediaItem = Sitecore.Context.Database.GetItem(dynamicLink.ItemId,
                    dynamicLink.Language ?? Sitecore.Context.Language);

                // Replace the tokens
                if (img.Attributes["alt"] != null && img.Attributes["alt"].Value.Trim() == AltToken)
                {
                    img.Attributes["alt"].Value = mediaItem.Alt;
                }

                if (img.Attributes["title"] != null && img.Attributes["title"].Value.Trim() == TitleToken)
                {
                    img.Attributes["title"].Value = mediaItem.Title;
                }
            }

            return document.DocumentNode.OuterHtml;
        }

        private string CleanImageSource(string imgSrcValue)
        {
            // First match the SHA1
            var shaMatcher = new Regex("\b[0-9a-f]{5,40}\b");
            var match = shaMatcher.Match(imgSrcValue);

            if (match.Success)
            {
                // Setup media url
                return string.Format("/~/media/{0}.ashx", match.Value);
            }

            return imgSrcValue;
        }
    }

 

Completion

Overall, this was a great feature that the client needed as they tried to make their website ADA compliant, by including the appropriate alt and title text values. I hope this helps you in your Sitecore development adventures.