Stratis DevEx: Introduction to the new developer experience for Stratis C# smart contracts
Introduction
Stratis DevEx is a recently accepted project to the Stratis Decentralized Accelerator program that is dedicated to significantly improving the developer experience for Stratis smart contract developers and achieving parity with the Ethereum smart contract developer experience and Ethereum projects and tools like Remix IDE and Solidity Visual Developer and Truffle VS Code. From the SDA proposal:
A key differentiator and strategic advantage of Stratis over other enterprise blockchain platforms like ConsenSys Quorum and Hyperledger Fabric is the strength of the tooling and eco-system and community around .NET and C#. Microsoft has invested decades and billions of dollars into making .NET one of the most developer-friendly and adopted enterprise technology platforms worldwide.
...
By integrating all the current tools and processes for smart contract development and security auditing into Visual Studio and supporting smart contract security techniques like formal verification, Stratis can leverage the investment and commitment Microsoft has made to tooling and developer experience and increase adoption of the Stratis smart contract and blockchain platform by enterprises and industries.
The main goals of Stratis DevEx are to integrate different static analysis processes and tools like Silver into IDEs like Visual Studio, and give the .NET smart contract developer visual feedback with features like Intellisense for correct smart contract code, code maps and visual interactive diagrams of smart contract code and projects, and static analysis of smart contract code to detect security and other issues.
We can look at the Stratis smart contract platform a bit and how developer tooling fits into the overall Stratis vision for .NET smart contracts for business.
Stratis Smart Contract Platform
Stratis is currently the only L1 blockchain and smart contract platform that is end-to-end .NET. The Stratis blockchain implementation is based on NBitcoin which is a full Bitcoin implementation for .NET while the Stratis smart contract platform uses the .NET CLR as its smart contract VM. This is in contrast to, for instance, the NEO blockchain platform which is a custom blockchain implemented in .NET and supporting the C# language for smart contracts, but which uses a custom-written virtual machine similar to how the Ethereum virtual machine is implemented.
In contrast to bespoke smart contract virtual machines like the NeoVM and EVM, the .NET CLR is a mature, widely used, 20+ year-old, language virtual machine technology that provides the runtime foundation for a huge swathe of the world's enterprise desktop and server and mobile apps. The open-source CoreCLR is the current generation implementation of the .NET CLR which succeeds the older .NET Framework CLR. The CLR provides many advanced language features like memory and type safety, support for basic object-oriented programming, and JIT code generation:
and is supported by a complete set of SDKs and libraries and compilers and tools and IDEs developed and supported by Microsoft.
Over the years the .NET CLR has also acquired several open-source tools and libraries for static analysis of CIL code including the widely used Cecil library. For developers, this means existing tools and libraries for static analysis of .NET CIL code can be re-used to analyze Stratis smart contract bytecode, in contrast to bespoke smart contract VMs where such tools must be newly developed and rely on a much smaller community of users.
From a security POV the CLR design has been studied and attacked for a long time. Microsoft obviously has a great deal invested in securing the .NET CLR against language-level security vulnerabilities and attempts to break its security model. The longevity of and sheer amount of resources dedicated to the .NET CLR has clear advantages for executing smart contracts designed for businesses and enterprises.
All the tools and methods for writing, debugging, testing, and transporting .NET code can be used to develop and deploy Stratis smart contracts. The main goal of Stratis DevEx is to extend the tremendous developer experience Microsoft has created around .NET with products and tools like Visual Studio and Roslyn, fully into developing .NET smart contracts and to provide enterprise .NET developers with the level of tooling they are accustomed to for smart contract development.
Adapting .NET for smart contracts
An obvious downside to using the .NET CLR and a language like C# for smart contracts is that most of the libraries and types and many of the language features available to .NET apps aren't going to be able to be used by smart contracts, due to determinism or other constraints. Classes that perform any kind of disk or network I/O obviously can't be used in smart contract code. .NET types that rely on dynamically allocated memory on the heap may fail to be constructed at runtime due to memory exhaustion on the node. This includes any language features that perform memory allocation like LINQ. Floating point types are inherently indeterministic and can't be used in smart contracts.
Certain language features are also problematic. Smart contracts use an explicit notion of state in contrast to regular programs where state is only implicitly defined, and things regular programs contain like mutable global variables or mutable public class fields or arbitrary objects allocated on the heap are generally not permitted in smart contracts. The only way to share or store state persistently in smart contract methods is by using the state store on the blockchain. Static properties are uninitialized non-deterministically in .NET and thus not allowed. Smart contracts allow the creation of custom data structures to be stored as records on the blockchain ledger, and while C# structs fit the bill of nested data structures in a smart contract class, certain features of C# structs like methods or static properties or multiple constructors or further-nested structs are not permitted. Certain C# language features like try-catch exception handling can also execute non-deterministically and thus can't be used in smart contracts.
In addition, a smart contract VM must have the ability to track the execution of smart contract code to do things like metering gas usage and monitoring memory usage. Execution of each smart contract instruction spends a fixed amount of gas up to a finite limit. In custom-built virtual machines like the EVM these abilities can be coded in, but the .NET CLR needs to be adapted to the requirements of smart contract execution.
To solve these problems Stratis relies on the processes of validation and rewriting. Smart contract validation refers to statically analyzing smart contract code to determine if the code satisfies all the constraints for executing as a valid smart contract. Rewriting is the process of injecting CIL instructions into a smart contract method to deterministically alter its behavior to support things like gas metering before it executes on the CLR. Let's look at these two processes a bit more, starting with rewriting.
Smart Contract Rewriting
The process of rewriting in Stratis refers to dynamically rewriting the CIL code of smart contract methods before execution to add needed functionality and constraints for execution as a blockchain smart contract method.
All the code for rewriting smart contracts can be found in the ILRewrite folder of the Stratis.SmartContracts.CLR project. Each rewriter class implements the IObserverMethodRewriter interface. An example is the GasInjectorRewriter rewriter:
/// <summary>
/// Rewrites a method to spend gas as execution occurs.
/// </summary>
public class GasInjectorRewriter : IObserverMethodRewriter
{
/// <inheritdoc />
public void Rewrite(MethodDefinition methodDefinition, ILProcessor il, ObserverRewriterContext context)
{
List<Instruction> branches = GetBranchingOps(methodDefinition).ToList();
List<Instruction> branchTos = branches.Select(x => (Instruction)x.Operand).ToList();
// Start from 2 because we setup Observer in first 2 instructions
int position = 2;
List<CodeSegment> segments = new List<CodeSegment>();
var codeSegment = new CodeSegment(methodDefinition);
while (position < methodDefinition.Body.Instructions.Count)
{
Instruction instruction =nmethodDefinition.Body.Instructions[position];
bool added = false;
// Start of a new segment. End last segment and start new one with this as the first instruction
if (branchTos.Contains(instruction))
{
if (codeSegment.Instructions.Any())
segments.Add(codeSegment);
codeSegment = new CodeSegment(methodDefinition);
codeSegment.Instructions.Add(instruction);
added = true;
}
// End of a segment. Add this as the last instruction and move onwards with a new segment
if (branches.Contains(instruction))
{
codeSegment.Instructions.Add(instruction);
segments.Add(codeSegment);
codeSegment = new CodeSegment(methodDefinition);
added = true;
}
// Just an in-between instruction. Add to current segment.
if (!added)
{
codeSegment.Instructions.Add(instruction);
}
position++;
}
// Got to end of the method. Add the last one if necessary
if (!segments.Contains(codeSegment) && codeSegment.Instructions.Any())
segments.Add(codeSegment);
foreach (CodeSegment segment in segments)
{
AddSpendGasMethodBeforeInstruction(il, context.Observer, context.ObserverVariable, segment);
}
...
This rewriter breaks up the CIL instructions in a method into basic blocks (called code segments in the Stratis code) separated by branching instructions. At the start of each code segment, CIL code is inserted that calls the gas meter for the cost of all the instructions in the segment.
A similar process happens in the MemoryLimitRewriter which rewrites CIL code to add instructions to track the size of arrays allocated in smart contract code and enforce deterministic limits.
By dynamically rewriting CIL code in smart contract methods before execution by the .NET CLR, Stratis can add capabilities like gas metering and memory limit checking not natively present in the CLR. This allows the .NET CLR to be used as a smart contract VM and for smart contract developers to benefit from all the advantages of a mature, battle-tested, widely-used language virtual machine.
Smart Contract CLR Validation
Smart contract validation in Stratis refers to the process of statically analyzing the types and language features that are used in .NET code to enforce constraints related to execution as a valid blockchain smart contract, such as determinism. Validation can happen either at the CIL bytecode level or the C# source code level. CIL code validation relies on the Cecil library while C# code validation relies on Roslyn.
CLR validation of smart contracts happens at compilation, and when a smart contract is uploaded and created through the node API. As a blockchain designed for business use Stratis doesn't currently allow unrestricted public smart contract execution on its Cirrus sidechain, the way Ethereum does for instance. Instead, contracts on the Cirus chain must be whitelisted via member voting by the InterFlux Decentralized Governance Board before they can be called. But all contracts passing through the smart contract node API must go through the CLR validation API.
We can look at this validation API a little and compare this to how the new DevEx tools implement some of these tasks at design time. The first milestone of the DevEx project is to implement this CLR validation as a Roslyn analyzer that can be used by IDEs like Visual Studio and Visual Studio Code to give the smart contract developer visual feedback on the validation status of his or her smart contract code at design time.
Smart Contact CLR Validation API
All the code for validation resides in the Stratis.SmartContracts.CLR.Validation project. This is the project that is linked to by the command-line validator tool. All the tool does is create instances of the SmartContractDeterminismValidator and SmartContractFormatValidator classes and run the specified smart contract assembly bytecode through the validators:
validationServiceResult.DeterminismValidationResult = new SctDeterminismValidator().Validate(moduleDefinition);
validationServiceResult.FormatValidationResult = new SmartContractFormatValidator().Validate(moduleDefinition.ModuleDefinition);
if (!validationServiceResult.DeterminismValidationResult.IsValid || !validationServiceResult.FormatValidationResult.IsValid)
console.WriteLine("Smart Contract failed validation. Run validate [FILE] for more info.");
Validation in Stratis occurs between validator classes and policy classes. Validator classes implement how the properties of CIL code are checked while policy classes implement declaring what CIL code properties are allowed in smart contract code. For instance, the Determinism Policy class looks like
public static class DeterminismPolicy
{
public static WhitelistPolicy WhitelistPolicy = new WhitelistPolicy()
.Namespace(nameof(System), AccessPolicy.Denied, SystemPolicy)
...
private static void SystemPolicy(NamespacePolicy policy)
{
foreach (Type type in Primitives.Types)
{
policy.Type(type, AccessPolicy.Allowed);
}
policy
.Type(typeof(Array).Name, AccessPolicy.Denied,
m => m.Member(nameof(Array.GetLength), AccessPolicy.Allowed)
.Member(nameof(Array.Copy), AccessPolicy.Allowed)
.Member(nameof(Array.GetValue), AccessPolicy.Allowed)
.Member(nameof(Array.SetValue), AccessPolicy.Allowed)
.Member(nameof(Array.Resize), AccessPolicy.Allowed))
.Type(typeof(void).Name, AccessPolicy.Allowed)
.Type(typeof(object).Name, AccessPolicy.Denied,
m => m.Member(nameof(ToString), AccessPolicy.Allowed)
.Constructor(AccessPolicy.Allowed));
...
This code, using a fluent style, expresses a policy for validating types that belong to the System namespace. The elements of the policy are, roughly:
Each type that belongs to the Primitives category (like
int,int64,byte, and so on) is allowed.Members of the
Arraytype are denied by default.Certain members of
Arraylike theGetLengthandCopymethods are allowed to be used.The
voidtype is allowed (like as the return type for a method).Members of the object type are not allowed, however certain members like the
ToStringmethod are allowed.
Validator classes rely on the Cecil library to analyze CIL code e.g. the SingleConstructorValidator:
using System.Collections.Generic;
using System.Linq;
using Mono.Cecil;
using Mono.Cecil.Rocks;
namespace Stratis.SmartContracts.CLR.Validation.Validators.Type
{
/// <summary>
/// Validates that a <see cref="Mono.Cecil.TypeDefinition"/> contains only a single constructor
/// </summary>
public class SingleConstructorValidator : ITypeDefinitionValidator
{
public const string MissingConstructorError = "Contract must define a constructor";
public const string SingleConstructorError = "Only a single constructor is allowed";
public IEnumerable<ValidationResult> Validate(TypeDefinition typeDef)
{
List<MethodDefinition> constructors = typeDef.GetConstructors()?.ToList();
if (constructors == null || !constructors.Any())
{
return new []
{
new TypeDefinitionValidationResult(MissingConstructorError)
};
}
if (constructors.Count > 1)
{
return new []
{
new TypeDefinitionValidationResult(SingleConstructorError)
};
}
return Enumerable.Empty<TypeDefinitionValidationResult>();
}
}
This code simply gets the list of constructors for a smart class in a Cecil ITypeDefinition, and returns an error if the count is not exactly one. This is just one out of a whole list of errors that are possible in the definition or format of a smart contract class and different validators are defined for each error that can be returned by the validator API.
Previously smart contract validation happened via a command-line tool: the sct tool which validated and compiled a C# smart contract and produced the bytecode which could then be uploaded to a node method. All the feedback from this tool is produced as console output. With the new developer experience, validation also happens when the developer writes code within a C# IDE like JetBrains Rider or Visual Studio Code with visual feedback in the IDE showing the location of each validation error, the details, severity and so on.
Smart Contract C# Validation
The main goal of the Stratis DevEx project is to improve the smart contract developer experience, especially for enterprise developers, and the .NET Compiler Platform or Roslyn is one of the pillars of this effort. One huge advantage of Roslyn is that Roslyn analyzers are supported in other IDEs besides Visual Studio, like JetBrains Rider and Visual Studio Code.
Stratis.CodeAnalysis Analyzer Installation
Most of the DevEx code will be delivered through NuGet packages from the Stratis DevEx public feed. To install the current version of the Stratis.CodeAnalysis analyzer, in your smart contract project's directory type:
dotnet add package Stratis.CodeAnalysis --version 0.1.0-ci88git68025ce --source https://www.myget.org/F/stratisdevex/api/v3/index.json
For Visual Studio and JetBrains Rider this should be sufficient to enable code analysis in your project. For Visual Studio Code you'll have to add an omnisharp.json file to your project with the setting:
{
"RoslynExtensionsOptions": {
"enableAnalyzersSupport": true
}
}
Alternatively, you can enable this setting at a user level. With Roslyn analyzers enabled you will be able to see smart contract code analysis and validation in Visual Studio Code:
Stratis.CodeAnalysis Analyzer Configuration
Note that if you install the analyzer at a solution level it will analyze all the projects in your solution in addition to the actual smart contract projects, which will likely result in a huge number of false positive errors. At the analyzer level, no information about solutions is available so an analyzer can't know that it's not analyzing a smart contract project and instead is analyzing a unit test project for instance. For analyzers that aren't first-party (like those shipped with the .NET SDK) it's probably a good idea to install them at the project level and not at the solution level but there is also an alternative for DevEx analyzers: using configuration files.
Create a file called stratisdev.cfg in your C# project. You must set the Build Action of this file to C# analyzer additional file: you can do this either in the MSBuild XML of your project file:
<ItemGroup>
<AdditionalFiles Include="stratisdev.cfg" />
</ItemGroup>
or in Visual Studio:
Inside the stratisdev.cfg file, add the following:
[Analyzer]
Enabled=False
Now when the analyzer runs it will first load the configuration in this file which tells it not to run analysis for this project. This file is loaded every time the compilation changes meaning every time it is changed, so editing and saving this file should trigger changes in the analyzer immediately.
Right now the only setting recognized is whether or not to enable the analyzer. The next version will recognize analyzer settings for enabling and disabling certain kinds of analysis for instance.
Stratis.CodeAnalysis Analyzer Internals
The validator derives from the Runtime class from the Stratis.DevEx.Base project. This class provides base initialization, logging, and configuration services for all the DevEx projects. The SmartContractAnalyzer class derives from the base DiagnosticAnalyzer class from which all Roslyn analyzers derive. The Initialize method of this class registers all the analyzer actions which respond to events that occur as the user makes edits to the compilation.
if (AnalyzerSetting(ctx.Compilation, "Analyzer", "Enabled", true))
{
...
ctx.RegisterSyntaxNodeAction(ctx => Validator.AnalyzeUsingDirective((UsingDirectiveSyntax)ctx.Node, ctx), SyntaxKind.UsingDirective);
ctx.RegisterSyntaxNodeAction(ctx => Validator.AnalyzeNamespaceDecl((NamespaceDeclarationSyntax)ctx.Node, ctx), SyntaxKind.NamespaceDeclaration);
...
There are two main categories of analyzer actions: syntax node actions, and operation actions. Syntax node actions correspond to purely syntactic changes to the document's AST while operation actions correspond to the results of semantic analysis of changes to the AST.
Analyzers are unit-tested using scaffolding created by the Analyzer project wizard. Code fragments are provided inline and the expected diagnostics are checked.
[TestMethod]
public async Task InvalidClassTypeDeclNotAllowedTest()
{
...
var code =
@" using Stratis.SmartContracts;
public class Player
{
public Player(ISmartContractState state, Address player, Address opponent, string gameName){}
public Player(ISmartContractState state, Address player){}
}
";
await VerifyCS.VerifyAnalyzerAsync(code, VerifyCS.Diagnostic("SC0022").WithSpan(2, 5, 7, 6).WithArguments("Player"));
} //Smart contract class can't have more than 1 constructor
CLR Validation vs. C# Validation
CLR validation occurs at compile-time or deploy-time by inspecting CIL code in bytecode assemblies using Cecil. C# validation occurs at design time in an IDE with visual feedback while the user edits code, using Roslyn. Even though intermediate languages typically are drastically different from high-level OOP languages like C#, the nature of the CLR as a virtual machine designed to support OOP languages means there is typically a 1-1 match between validation cases from CIL to C# e.g. we saw in the SingleConstructorValidator above how we can use Cecil's ITypeDefinition to examine and validate class constructors in CIL code. To do the same for C# source code in Roslyn we can say
public static Diagnostic AnalyzeClassDecl(ClassDeclarationSyntax node, SemanticModel model)
{
var type = model.GetDeclaredSymbol(node) as ITypeSymbol;
var identifier = node.ChildTokens().First(t => t.IsKind(SyntaxKind.IdentifierToken));
Debug("Class {0} declared at {1}.", type.ToDisplayString(), node.GetLineLocation());
var cdecls = node.ChildNodes().OfType<ConstructorDeclarationSyntax>();
if (cdecls.Count() > 1)
{
return CreateDiagnostic("SC0022", node.GetLocation(), identifier.Text);
}
...
The AnalyzeDecl method is a static method of our Stratis.CodeAnalysis Validator class. This method is registered in our SmartContractAnalyzer class as an action when Roslyn encounters or updates a C# class declaration:
ctx.RegisterSyntaxNodeAction(ctx => AnalyzeClassDecl((ClassDeclarationSyntax)ctx.Node, ctx), SyntaxKind.ClassDeclaration);
In this method, we use LINQ to get child syntax nodes of the class declaration node that are constructor declarations. If there is more than 1, we emit a diagnostic describing the problem with the location of the class, together with its name.
Conclusion
The Stratis.CodeAnalysis analyzer moves the CLR validation process from deploy time to design time with immediate visual feedback on validation errors, and provides a significantly improved developer experience for smart contract development.
The traditional way to provide an enhanced developer experience for a .NET API or tool in Visual Studio is to develop a Visual Studio extension. This is a complex undertaking that is not portable to other environments. Visual Studio Code and JetBrains Rider both have their own extensibility models and languages, namely JavaScript/TypeScript and Java.
The Roslyn analyzer environment is a lot saner and simpler than the general Visual Studio extension environment. There are a lot of advantages to using Roslyn analyzers for enhancing the developer experience for smart contracts, and the Roslyn analyzer lifecycle is flexible enough to accommodate sending data to external apps as the user edits code and compiles their .NET project.
Moving GUI operations like displaying smart contract code maps, diagrams, and disassembly or providing an interface for deployment and execution, to a portable application window server that can be called externally by analyzers running in different IDEs may have a lot of advantages, not the least of which is code portability and maintainability. Future articles will explore this use of analyzers to create true portable IDE extensions for smart contract development.